2023-04-05 09:53:31,193 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740 2023-04-05 09:53:31,206 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-05 09:53:31,221 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-05 09:53:31,222 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a, deleteOnExit=true 2023-04-05 09:53:31,222 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-05 09:53:31,222 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/test.cache.data in system properties and HBase conf 2023-04-05 09:53:31,223 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/hadoop.tmp.dir in system properties and HBase conf 2023-04-05 09:53:31,223 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/hadoop.log.dir in system properties and HBase conf 2023-04-05 09:53:31,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-05 09:53:31,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-05 09:53:31,224 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-05 09:53:31,342 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-05 09:53:31,762 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-05 09:53:31,768 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-05 09:53:31,768 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-05 09:53:31,769 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-05 09:53:31,769 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 09:53:31,770 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-05 09:53:31,770 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-05 09:53:31,771 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 09:53:31,771 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 09:53:31,772 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-05 09:53:31,772 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/nfs.dump.dir in system properties and HBase conf 2023-04-05 09:53:31,773 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/java.io.tmpdir in system properties and HBase conf 2023-04-05 09:53:31,773 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 09:53:31,773 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-05 09:53:31,774 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-05 09:53:32,388 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 09:53:32,392 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 09:53:32,708 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-05 09:53:32,888 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-05 09:53:32,906 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 09:53:32,947 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 09:53:32,979 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/java.io.tmpdir/Jetty_localhost_38585_hdfs____qk2gwn/webapp 2023-04-05 09:53:33,143 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38585 2023-04-05 09:53:33,180 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 09:53:33,180 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 09:53:33,606 WARN [Listener at localhost/45259] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 09:53:33,764 WARN [Listener at localhost/45259] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-05 09:53:33,786 WARN [Listener at localhost/45259] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 09:53:33,794 INFO [Listener at localhost/45259] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 09:53:33,799 INFO [Listener at localhost/45259] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/java.io.tmpdir/Jetty_localhost_43163_datanode____j1sbw6/webapp 2023-04-05 09:53:33,903 INFO [Listener at localhost/45259] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43163 2023-04-05 09:53:34,232 WARN [Listener at localhost/42121] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 09:53:34,732 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52e1b445e3a1fe4f: Processing first storage report for DS-b5b36e05-276b-4ccb-9417-d46e322efe13 from datanode ee8eb019-bae7-4188-8cbb-c2ed540bd4d2 2023-04-05 09:53:34,734 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52e1b445e3a1fe4f: from storage DS-b5b36e05-276b-4ccb-9417-d46e322efe13 node DatanodeRegistration(127.0.0.1:35759, datanodeUuid=ee8eb019-bae7-4188-8cbb-c2ed540bd4d2, infoPort=38093, infoSecurePort=0, ipcPort=42121, storageInfo=lv=-57;cid=testClusterID;nsid=324615496;c=1680688412466), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-05 09:53:34,734 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52e1b445e3a1fe4f: Processing first storage report for DS-5dd91a68-e6fa-4d80-9a39-e000290d393d from datanode ee8eb019-bae7-4188-8cbb-c2ed540bd4d2 2023-04-05 09:53:34,735 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52e1b445e3a1fe4f: from storage DS-5dd91a68-e6fa-4d80-9a39-e000290d393d node DatanodeRegistration(127.0.0.1:35759, datanodeUuid=ee8eb019-bae7-4188-8cbb-c2ed540bd4d2, infoPort=38093, infoSecurePort=0, ipcPort=42121, storageInfo=lv=-57;cid=testClusterID;nsid=324615496;c=1680688412466), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-05 09:53:34,760 DEBUG [Listener at localhost/42121] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740 2023-04-05 09:53:34,824 INFO [Listener at localhost/42121] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a/zookeeper_0, clientPort=62753, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-05 09:53:34,840 INFO [Listener at localhost/42121] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62753 2023-04-05 09:53:34,848 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:34,851 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:35,539 INFO [Listener at localhost/42121] util.FSUtils(471): Created version file at hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8 with version=8 2023-04-05 09:53:35,539 INFO [Listener at localhost/42121] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/hbase-staging 2023-04-05 09:53:35,874 INFO [Listener at localhost/42121] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-05 09:53:36,454 INFO [Listener at localhost/42121] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-05 09:53:36,486 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,486 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,486 INFO [Listener at localhost/42121] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 09:53:36,486 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,487 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 09:53:36,628 INFO [Listener at localhost/42121] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 09:53:36,702 DEBUG [Listener at localhost/42121] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-05 09:53:36,796 INFO [Listener at localhost/42121] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34901 2023-04-05 09:53:36,806 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:36,808 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:36,829 INFO [Listener at localhost/42121] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34901 connecting to ZooKeeper ensemble=127.0.0.1:62753 2023-04-05 09:53:36,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:349010x0, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 09:53:36,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34901-0x10153bdea1d0000 connected 2023-04-05 09:53:36,900 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 09:53:36,901 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 09:53:36,905 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 09:53:36,914 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34901 2023-04-05 09:53:36,915 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34901 2023-04-05 09:53:36,915 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34901 2023-04-05 09:53:36,915 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34901 2023-04-05 09:53:36,916 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34901 2023-04-05 09:53:36,922 INFO [Listener at localhost/42121] master.HMaster(444): hbase.rootdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8, hbase.cluster.distributed=false 2023-04-05 09:53:36,991 INFO [Listener at localhost/42121] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-05 09:53:36,992 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,992 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,992 INFO [Listener at localhost/42121] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 09:53:36,992 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 09:53:36,992 INFO [Listener at localhost/42121] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 09:53:36,997 INFO [Listener at localhost/42121] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 09:53:37,000 INFO [Listener at localhost/42121] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46827 2023-04-05 09:53:37,002 INFO [Listener at localhost/42121] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-05 09:53:37,008 DEBUG [Listener at localhost/42121] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-05 09:53:37,009 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:37,012 INFO [Listener at localhost/42121] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:37,015 INFO [Listener at localhost/42121] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46827 connecting to ZooKeeper ensemble=127.0.0.1:62753 2023-04-05 09:53:37,019 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:468270x0, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 09:53:37,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46827-0x10153bdea1d0001 connected 2023-04-05 09:53:37,020 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 09:53:37,021 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 09:53:37,022 DEBUG [Listener at localhost/42121] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 09:53:37,023 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46827 2023-04-05 09:53:37,023 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46827 2023-04-05 09:53:37,024 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46827 2023-04-05 09:53:37,028 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46827 2023-04-05 09:53:37,028 DEBUG [Listener at localhost/42121] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46827 2023-04-05 09:53:37,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:37,040 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 09:53:37,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:37,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 09:53:37,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 09:53:37,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:37,061 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 09:53:37,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34901,1680688415681 from backup master directory 2023-04-05 09:53:37,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 09:53:37,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:37,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 09:53:37,067 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 09:53:37,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:37,070 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-05 09:53:37,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-05 09:53:37,159 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/hbase.id with ID: 36220915-8569-4221-a71f-0b0105f844d6 2023-04-05 09:53:37,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 09:53:37,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:37,251 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x67c1b592 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:37,283 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@265fee6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:37,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:53:37,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-05 09:53:37,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-05 09:53:37,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-05 09:53:37,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:37,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:37,331 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 09:53:37,363 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store-tmp 2023-04-05 09:53:37,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:37,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 09:53:37,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:53:37,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:53:37,792 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 09:53:37,792 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:53:37,792 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:53:37,792 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 09:53:37,794 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/WALs/jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:37,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34901%2C1680688415681, suffix=, logDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/WALs/jenkins-hbase4.apache.org,34901,1680688415681, archiveDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/oldWALs, maxLogs=10 2023-04-05 09:53:37,867 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK] 2023-04-05 09:53:37,875 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:37,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/WALs/jenkins-hbase4.apache.org,34901,1680688415681/jenkins-hbase4.apache.org%2C34901%2C1680688415681.1680688417826 2023-04-05 09:53:37,932 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK]] 2023-04-05 09:53:37,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:53:37,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:37,937 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:37,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:37,994 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:38,001 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-05 09:53:38,024 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-05 09:53:38,036 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,042 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:38,044 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:38,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 09:53:38,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:53:38,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67551379, jitterRate=0.006593987345695496}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 09:53:38,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 09:53:38,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-05 09:53:38,084 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-05 09:53:38,084 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-05 09:53:38,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-05 09:53:38,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-05 09:53:38,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-05 09:53:38,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-05 09:53:38,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-05 09:53:38,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-05 09:53:38,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-05 09:53:38,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-05 09:53:38,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-05 09:53:38,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-05 09:53:38,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-05 09:53:38,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:38,203 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-05 09:53:38,204 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-05 09:53:38,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-05 09:53:38,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 09:53:38,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 09:53:38,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:38,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34901,1680688415681, sessionid=0x10153bdea1d0000, setting cluster-up flag (Was=false) 2023-04-05 09:53:38,240 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:38,247 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-05 09:53:38,248 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:38,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:38,258 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-05 09:53:38,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:38,262 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.hbase-snapshot/.tmp 2023-04-05 09:53:38,333 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(952): ClusterId : 36220915-8569-4221-a71f-0b0105f844d6 2023-04-05 09:53:38,337 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-05 09:53:38,341 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-05 09:53:38,341 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-05 09:53:38,345 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-05 09:53:38,346 DEBUG [RS:0;jenkins-hbase4:46827] zookeeper.ReadOnlyZKClient(139): Connect 0x479f9e9d to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:38,355 DEBUG [RS:0;jenkins-hbase4:46827] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4958781e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:38,355 DEBUG [RS:0;jenkins-hbase4:46827] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d67f5ed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-05 09:53:38,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-05 09:53:38,381 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46827 2023-04-05 09:53:38,383 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 09:53:38,383 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 09:53:38,383 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 09:53:38,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 09:53:38,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-05 09:53:38,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-05 09:53:38,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680688448385 2023-04-05 09:53:38,385 INFO [RS:0;jenkins-hbase4:46827] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-05 09:53:38,386 INFO [RS:0;jenkins-hbase4:46827] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-05 09:53:38,386 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1023): About to register with Master. 2023-04-05 09:53:38,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-05 09:53:38,389 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34901,1680688415681 with isa=jenkins-hbase4.apache.org/172.31.14.131:46827, startcode=1680688416991 2023-04-05 09:53:38,391 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 09:53:38,391 INFO [PEWorker-2] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-05 09:53:38,397 INFO [PEWorker-2] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 09:53:38,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-05 09:53:38,406 DEBUG [RS:0;jenkins-hbase4:46827] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-05 09:53:38,408 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-05 09:53:38,408 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-05 09:53:38,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-05 09:53:38,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-05 09:53:38,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-05 09:53:38,413 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-05 09:53:38,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-05 09:53:38,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-05 09:53:38,417 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-05 09:53:38,419 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680688418419,5,FailOnTimeoutGroup] 2023-04-05 09:53:38,419 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680688418419,5,FailOnTimeoutGroup] 2023-04-05 09:53:38,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-05 09:53:38,421 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,436 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 09:53:38,437 INFO [PEWorker-2] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 09:53:38,438 INFO [PEWorker-2] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8 2023-04-05 09:53:38,450 DEBUG [PEWorker-2] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:38,453 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 09:53:38,456 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/info 2023-04-05 09:53:38,457 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 09:53:38,458 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,458 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 09:53:38,461 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/rep_barrier 2023-04-05 09:53:38,462 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 09:53:38,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,463 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 09:53:38,464 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35993, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-05 09:53:38,465 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/table 2023-04-05 09:53:38,466 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 09:53:38,466 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,468 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740 2023-04-05 09:53:38,469 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740 2023-04-05 09:53:38,473 DEBUG [PEWorker-2] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:53:38,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,475 DEBUG [PEWorker-2] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 09:53:38,478 DEBUG [PEWorker-2] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:53:38,479 INFO [PEWorker-2] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61239285, jitterRate=-0.08746354281902313}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:53:38,480 DEBUG [PEWorker-2] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 09:53:38,480 DEBUG [PEWorker-2] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 09:53:38,480 INFO [PEWorker-2] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 09:53:38,480 DEBUG [PEWorker-2] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 09:53:38,480 DEBUG [PEWorker-2] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 09:53:38,480 DEBUG [PEWorker-2] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 09:53:38,481 INFO [PEWorker-2] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 09:53:38,481 DEBUG [PEWorker-2] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 09:53:38,485 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 09:53:38,486 INFO [PEWorker-2] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-05 09:53:38,491 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8 2023-04-05 09:53:38,491 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45259 2023-04-05 09:53:38,491 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-05 09:53:38,494 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-05 09:53:38,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 09:53:38,497 DEBUG [RS:0;jenkins-hbase4:46827] zookeeper.ZKUtil(162): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,497 WARN [RS:0;jenkins-hbase4:46827] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 09:53:38,497 INFO [RS:0;jenkins-hbase4:46827] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 09:53:38,498 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/WALs/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,499 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46827,1680688416991] 2023-04-05 09:53:38,507 DEBUG [RS:0;jenkins-hbase4:46827] zookeeper.ZKUtil(162): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,507 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-05 09:53:38,510 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-05 09:53:38,517 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-05 09:53:38,527 INFO [RS:0;jenkins-hbase4:46827] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-05 09:53:38,545 INFO [RS:0;jenkins-hbase4:46827] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-05 09:53:38,549 INFO [RS:0;jenkins-hbase4:46827] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-05 09:53:38,549 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,550 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-05 09:53:38,557 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,557 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,558 DEBUG [RS:0;jenkins-hbase4:46827] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 09:53:38,559 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,559 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,559 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,574 INFO [RS:0;jenkins-hbase4:46827] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-05 09:53:38,576 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46827,1680688416991-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:38,591 INFO [RS:0;jenkins-hbase4:46827] regionserver.Replication(203): jenkins-hbase4.apache.org,46827,1680688416991 started 2023-04-05 09:53:38,591 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46827,1680688416991, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46827, sessionid=0x10153bdea1d0001 2023-04-05 09:53:38,592 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-05 09:53:38,592 DEBUG [RS:0;jenkins-hbase4:46827] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,592 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46827,1680688416991' 2023-04-05 09:53:38,592 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46827,1680688416991' 2023-04-05 09:53:38,593 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-05 09:53:38,594 DEBUG [RS:0;jenkins-hbase4:46827] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-05 09:53:38,594 DEBUG [RS:0;jenkins-hbase4:46827] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-05 09:53:38,594 INFO [RS:0;jenkins-hbase4:46827] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-05 09:53:38,594 INFO [RS:0;jenkins-hbase4:46827] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-05 09:53:38,661 DEBUG [jenkins-hbase4:34901] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-05 09:53:38,664 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46827,1680688416991, state=OPENING 2023-04-05 09:53:38,671 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-05 09:53:38,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:38,674 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 09:53:38,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:53:38,705 INFO [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46827%2C1680688416991, suffix=, logDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/WALs/jenkins-hbase4.apache.org,46827,1680688416991, archiveDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/oldWALs, maxLogs=32 2023-04-05 09:53:38,724 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK] 2023-04-05 09:53:38,728 INFO [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/WALs/jenkins-hbase4.apache.org,46827,1680688416991/jenkins-hbase4.apache.org%2C46827%2C1680688416991.1680688418708 2023-04-05 09:53:38,728 DEBUG [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK]] 2023-04-05 09:53:38,863 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:38,865 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-05 09:53:38,869 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-05 09:53:38,882 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-05 09:53:38,883 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 09:53:38,887 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46827%2C1680688416991.meta, suffix=.meta, logDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/WALs/jenkins-hbase4.apache.org,46827,1680688416991, archiveDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/oldWALs, maxLogs=32 2023-04-05 09:53:38,910 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK] 2023-04-05 09:53:38,915 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/WALs/jenkins-hbase4.apache.org,46827,1680688416991/jenkins-hbase4.apache.org%2C46827%2C1680688416991.meta.1680688418889.meta 2023-04-05 09:53:38,916 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35759,DS-b5b36e05-276b-4ccb-9417-d46e322efe13,DISK]] 2023-04-05 09:53:38,916 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:53:38,918 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-05 09:53:38,940 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-05 09:53:38,945 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-05 09:53:38,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-05 09:53:38,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:38,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-05 09:53:38,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-05 09:53:38,957 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 09:53:38,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/info 2023-04-05 09:53:38,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/info 2023-04-05 09:53:38,959 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 09:53:38,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 09:53:38,962 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/rep_barrier 2023-04-05 09:53:38,962 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/rep_barrier 2023-04-05 09:53:38,962 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 09:53:38,963 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,963 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 09:53:38,965 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/table 2023-04-05 09:53:38,965 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/table 2023-04-05 09:53:38,965 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 09:53:38,966 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:38,968 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740 2023-04-05 09:53:38,971 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740 2023-04-05 09:53:38,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:53:38,977 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 09:53:38,979 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64377793, jitterRate=-0.04069612920284271}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:53:38,979 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 09:53:38,988 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680688418855 2023-04-05 09:53:39,005 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-05 09:53:39,005 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-05 09:53:39,006 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46827,1680688416991, state=OPEN 2023-04-05 09:53:39,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-05 09:53:39,009 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 09:53:39,013 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-05 09:53:39,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46827,1680688416991 in 332 msec 2023-04-05 09:53:39,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-05 09:53:39,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 521 msec 2023-04-05 09:53:39,025 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 718 msec 2023-04-05 09:53:39,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680688419025, completionTime=-1 2023-04-05 09:53:39,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-05 09:53:39,025 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-05 09:53:39,103 DEBUG [hconnection-0x65923c37-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:39,107 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:39,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-05 09:53:39,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680688479126 2023-04-05 09:53:39,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680688539126 2023-04-05 09:53:39,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 100 msec 2023-04-05 09:53:39,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34901,1680688415681-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:39,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34901,1680688415681-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:39,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34901,1680688415681-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:39,157 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34901, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:39,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-05 09:53:39,162 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-05 09:53:39,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-05 09:53:39,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 09:53:39,185 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-05 09:53:39,188 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:53:39,191 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:53:39,219 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,222 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570 empty. 2023-04-05 09:53:39,223 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,223 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-05 09:53:39,253 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-05 09:53:39,256 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d0f966b2d8313b2cb7f9ce12df1ff570, NAME => 'hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d0f966b2d8313b2cb7f9ce12df1ff570, disabling compactions & flushes 2023-04-05 09:53:39,272 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. after waiting 0 ms 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,272 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,272 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d0f966b2d8313b2cb7f9ce12df1ff570: 2023-04-05 09:53:39,279 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:53:39,299 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680688419283"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688419283"}]},"ts":"1680688419283"} 2023-04-05 09:53:39,334 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:53:39,337 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:53:39,343 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688419337"}]},"ts":"1680688419337"} 2023-04-05 09:53:39,349 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-05 09:53:39,360 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0f966b2d8313b2cb7f9ce12df1ff570, ASSIGN}] 2023-04-05 09:53:39,362 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0f966b2d8313b2cb7f9ce12df1ff570, ASSIGN 2023-04-05 09:53:39,364 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d0f966b2d8313b2cb7f9ce12df1ff570, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:53:39,515 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0f966b2d8313b2cb7f9ce12df1ff570, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:39,516 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680688419515"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688419515"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688419515"}]},"ts":"1680688419515"} 2023-04-05 09:53:39,522 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d0f966b2d8313b2cb7f9ce12df1ff570, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:53:39,683 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d0f966b2d8313b2cb7f9ce12df1ff570, NAME => 'hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:53:39,686 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,687 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:39,687 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,687 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,690 INFO [StoreOpener-d0f966b2d8313b2cb7f9ce12df1ff570-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,693 DEBUG [StoreOpener-d0f966b2d8313b2cb7f9ce12df1ff570-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/info 2023-04-05 09:53:39,693 DEBUG [StoreOpener-d0f966b2d8313b2cb7f9ce12df1ff570-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/info 2023-04-05 09:53:39,694 INFO [StoreOpener-d0f966b2d8313b2cb7f9ce12df1ff570-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d0f966b2d8313b2cb7f9ce12df1ff570 columnFamilyName info 2023-04-05 09:53:39,695 INFO [StoreOpener-d0f966b2d8313b2cb7f9ce12df1ff570-1] regionserver.HStore(310): Store=d0f966b2d8313b2cb7f9ce12df1ff570/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:39,696 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,697 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:53:39,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:53:39,707 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d0f966b2d8313b2cb7f9ce12df1ff570; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59287705, jitterRate=-0.11654435098171234}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 09:53:39,707 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d0f966b2d8313b2cb7f9ce12df1ff570: 2023-04-05 09:53:39,709 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570., pid=6, masterSystemTime=1680688419676 2023-04-05 09:53:39,714 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,714 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:53:39,716 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0f966b2d8313b2cb7f9ce12df1ff570, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:39,716 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680688419715"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688419715"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688419715"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688419715"}]},"ts":"1680688419715"} 2023-04-05 09:53:39,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-05 09:53:39,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d0f966b2d8313b2cb7f9ce12df1ff570, server=jenkins-hbase4.apache.org,46827,1680688416991 in 198 msec 2023-04-05 09:53:39,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-05 09:53:39,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d0f966b2d8313b2cb7f9ce12df1ff570, ASSIGN in 365 msec 2023-04-05 09:53:39,729 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:53:39,730 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688419730"}]},"ts":"1680688419730"} 2023-04-05 09:53:39,733 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-05 09:53:39,736 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:53:39,738 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 561 msec 2023-04-05 09:53:39,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-05 09:53:39,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-05 09:53:39,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:53:39,824 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-05 09:53:39,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 09:53:39,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-04-05 09:53:39,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-05 09:53:39,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 09:53:39,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-05 09:53:39,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-05 09:53:39,898 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-05 09:53:39,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.831sec 2023-04-05 09:53:39,900 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-05 09:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-05 09:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-05 09:53:39,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34901,1680688415681-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-05 09:53:39,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34901,1680688415681-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-05 09:53:39,914 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-05 09:53:39,938 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:39,942 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:39,954 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:39,966 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:39,976 INFO [Listener at localhost/42121] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:53:39,996 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=169, AvailableMemoryMB=2075 2023-04-05 09:53:40,013 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:53:40,016 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:53:40,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:53:40,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:53:40,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:53:40,035 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:53:40,037 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:53:40,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-05 09:53:40,040 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,041 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 empty. 2023-04-05 09:53:40,043 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,043 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:53:40,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 09:53:40,063 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:53:40,065 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6f2b27e5e74b06149aed9138d7651e34, NAME => 'TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6f2b27e5e74b06149aed9138d7651e34, disabling compactions & flushes 2023-04-05 09:53:40,076 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. after waiting 0 ms 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,076 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,076 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:40,080 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:53:40,081 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688420081"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688420081"}]},"ts":"1680688420081"} 2023-04-05 09:53:40,084 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:53:40,086 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:53:40,086 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688420086"}]},"ts":"1680688420086"} 2023-04-05 09:53:40,088 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:53:40,091 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, ASSIGN}] 2023-04-05 09:53:40,093 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, ASSIGN 2023-04-05 09:53:40,095 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:53:40,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 09:53:40,246 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6f2b27e5e74b06149aed9138d7651e34, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:40,246 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688420246"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688420246"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688420246"}]},"ts":"1680688420246"} 2023-04-05 09:53:40,250 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:53:40,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 09:53:40,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6f2b27e5e74b06149aed9138d7651e34, NAME => 'TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:53:40,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:53:40,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,414 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,416 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A 2023-04-05 09:53:40,416 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A 2023-04-05 09:53:40,417 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:53:40,418 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2b27e5e74b06149aed9138d7651e34 columnFamilyName A 2023-04-05 09:53:40,419 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(310): Store=6f2b27e5e74b06149aed9138d7651e34/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:40,419 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,421 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B 2023-04-05 09:53:40,421 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B 2023-04-05 09:53:40,421 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:53:40,422 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2b27e5e74b06149aed9138d7651e34 columnFamilyName B 2023-04-05 09:53:40,423 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(310): Store=6f2b27e5e74b06149aed9138d7651e34/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:40,423 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,425 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C 2023-04-05 09:53:40,425 DEBUG [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C 2023-04-05 09:53:40,426 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:53:40,426 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2b27e5e74b06149aed9138d7651e34 columnFamilyName C 2023-04-05 09:53:40,427 INFO [StoreOpener-6f2b27e5e74b06149aed9138d7651e34-1] regionserver.HStore(310): Store=6f2b27e5e74b06149aed9138d7651e34/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:53:40,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:53:40,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:53:40,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6f2b27e5e74b06149aed9138d7651e34; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58871067, jitterRate=-0.12275274097919464}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:53:40,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:40,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., pid=11, masterSystemTime=1680688420404 2023-04-05 09:53:40,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,445 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6f2b27e5e74b06149aed9138d7651e34, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:40,445 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688420444"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688420444"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688420444"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688420444"}]},"ts":"1680688420444"} 2023-04-05 09:53:40,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-05 09:53:40,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 in 197 msec 2023-04-05 09:53:40,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-05 09:53:40,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, ASSIGN in 359 msec 2023-04-05 09:53:40,455 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:53:40,455 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688420455"}]},"ts":"1680688420455"} 2023-04-05 09:53:40,458 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:53:40,461 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:53:40,463 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 430 msec 2023-04-05 09:53:40,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 09:53:40,661 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-05 09:53:40,701 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x44ef43e3 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,706 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2678d495, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,713 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,717 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,722 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:53:40,724 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:53:40,733 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,738 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b886c8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,738 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,743 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,743 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,750 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,751 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,756 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,756 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,760 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,761 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,769 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,770 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,795 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,796 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,801 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,801 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:53:40,807 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:53:40,821 DEBUG [hconnection-0x50724ede-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,821 DEBUG [hconnection-0x217a24de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,823 DEBUG [hconnection-0x1531940c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,826 DEBUG [hconnection-0x7e007515-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,827 DEBUG [hconnection-0x5ab73287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,829 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33022, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,831 DEBUG [hconnection-0x5dfec272-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,831 DEBUG [hconnection-0x9312249-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,832 DEBUG [hconnection-0x3781157c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,832 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,840 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,842 DEBUG [hconnection-0x224073f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:53:40,842 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,843 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,850 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:40,857 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,861 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:40,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:40,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:40,877 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:53:40,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:40,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:40,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:40,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:40,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:40,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:40,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:40,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:40,885 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:40,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:40,888 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:40,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:40,910 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:40,913 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:40,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:40,914 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:40,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:40,927 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:40,927 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:40,928 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:40,928 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:40,928 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:40,928 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:41,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688420977, max timestamp=1680688421018Num uniques 859; 2023-04-05 09:53:41,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:53:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688420986, max timestamp=1680688421040Num uniques 859; 2023-04-05 09:53:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688420986, max timestamp=1680688421018Num uniques 859; 2023-04-05 09:53:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 09:53:41,094 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/32794aceae3c4bac85cf05ff2bfd3984 2023-04-05 09:53:41,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688481161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688481163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688481163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688481170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688481173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688481321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688481321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688481322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688481322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688481322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688481527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688481529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688481529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688481531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688481534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,624 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/55c18415f39546b08dc3bf5884f77cb2 2023-04-05 09:53:41,741 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/d3e50ab59b0841478c7f59487002522e 2023-04-05 09:53:41,753 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/32794aceae3c4bac85cf05ff2bfd3984 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984 2023-04-05 09:53:41,765 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 09:53:41,770 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 09:53:41,777 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/55c18415f39546b08dc3bf5884f77cb2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2 2023-04-05 09:53:41,795 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 09:53:41,798 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/d3e50ab59b0841478c7f59487002522e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e 2023-04-05 09:53:41,820 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 09:53:41,823 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for 6f2b27e5e74b06149aed9138d7651e34 in 909ms, sequenceid=11, compaction requested=false 2023-04-05 09:53:41,825 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 09:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:41,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:41,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:41,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:41,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:41,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=140.49 KB heapSize=399.30 KB 2023-04-05 09:53:41,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:41,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:41,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:41,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,846 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:41,846 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:41,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0dcd9f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:41,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,853 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:41,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:41,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:41,860 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:41,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:41,869 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:41,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:41,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:41,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:41,870 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:41,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:41,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:41,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:41,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,883 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688481885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688481899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688481900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688481902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:41,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688481903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,956 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:41,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:41,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:41,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:41,964 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:41,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:41,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:41,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:41,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:41,966 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:41,967 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:42,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688482006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688482007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688482007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688482008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688482019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,067 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:42,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:42,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:42,086 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:42,086 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:42,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:42,090 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,094 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,094 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,095 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,095 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:42,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:42,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:42,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:42,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74798dd0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:42,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:42,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:42,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:42,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:42,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:42,122 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,122 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,122 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:42,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:42,131 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:42,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:42,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:42,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:42,131 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:42,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:42,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:42,156 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:42,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:42,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:42,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688482214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688482214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688482216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688482216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688482224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,260 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:42,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:42,276 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:42,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,280 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:42,280 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:42,281 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,281 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,284 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,285 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,286 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:42,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:42,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:42,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:42,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:42,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:42,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:42,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:42,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@583fb4c7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:42,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:42,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:42,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:42,306 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:42,305 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:42,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:42,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:42,322 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:42,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:42,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:42,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:42,322 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:42,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,326 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,326 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,331 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:42,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:42,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:42,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:42,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/1d8546bd0fc84443a6eff937dfc8c929 2023-04-05 09:53:42,442 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:42,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:42,453 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:42,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:42,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:42,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,459 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,459 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:42,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:42,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:42,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:42,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:42,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:42,468 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,468 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:42,468 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:42,468 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:42,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:42,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:42,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:42,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:42,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581e9e89[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:42,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:42,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:42,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:42,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:42,491 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:42,491 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:42,491 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:42,491 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:42,491 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:42,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,515 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:42,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:42,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,518 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,518 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,518 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:42,518 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:42,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688482520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688482520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688482521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688482523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:42,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688482532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,619 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:42,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:42,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:42,638 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:42,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:42,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:42,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,641 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:42,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:42,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:42,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:42,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:42,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:42,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:42,668 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,668 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:42,668 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:42,668 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,672 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:42,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cdbd120[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:42,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:42,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,678 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:42,678 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:42,678 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:42,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:42,681 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:42,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,681 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:42,695 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:42,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:42,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:42,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:42,695 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:42,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:42,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:42,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:42,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:42,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,717 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/c656266c7239417bbc8aaebe9af6cde6 2023-04-05 09:53:42,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:42,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:42,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:42,832 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:42,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:42,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:42,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:42,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:42,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:42,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:42,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:42,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:42,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:42,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:42,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:42,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:42,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:42,912 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:42,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/17f05006c4744dd5bd4b1db1511dfc15 2023-04-05 09:53:42,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:42,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4c466[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:42,966 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:42,966 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,968 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:42,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:42,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:42,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:42,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:42,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:42,987 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:42,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:42,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:42,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:42,987 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:42,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/1d8546bd0fc84443a6eff937dfc8c929 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929 2023-04-05 09:53:42,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:42,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:42,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:42,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929, entries=200, sequenceid=48, filesize=13.9 K 2023-04-05 09:53:43,001 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:43,001 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:43,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:43,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:43,001 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:43,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:43,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/c656266c7239417bbc8aaebe9af6cde6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6 2023-04-05 09:53:43,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:43,001 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:43,009 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:43,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:43,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:43,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:43,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:43,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:43,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688483035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688483035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688483036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688483037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688483041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 09:53:43,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/17f05006c4744dd5bd4b1db1511dfc15 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15 2023-04-05 09:53:43,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 09:53:43,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.49 KB/143865, heapSize ~399.26 KB/408840, currentSize=46.96 KB/48090 for 6f2b27e5e74b06149aed9138d7651e34 in 1230ms, sequenceid=48, compaction requested=false 2023-04-05 09:53:43,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:43,114 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:43,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:43,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:43,132 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:43,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:43,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:43,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:43,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:43,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:43,139 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:43,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:43,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:43,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:43,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:43,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:43,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:43,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:43,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:43,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:43,145 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:43,147 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:43,147 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:43,147 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:43,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:43,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:43,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:43,614 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/7cd045e6b18e4189a6edf00123edd056 2023-04-05 09:53:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:44,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:44,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688424064, max timestamp=1680688424072Num uniques 859; 2023-04-05 09:53:44,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688424064, max timestamp=1680688424071Num uniques 859; 2023-04-05 09:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:44,081 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/41a2d84475d3468285bfceeb3c3bd85f 2023-04-05 09:53:44,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688424064, max timestamp=1680688424072Num uniques 859; 2023-04-05 09:53:44,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 09:53:44,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688484143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688484146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688484141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688484154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688424103, max timestamp=1680688424151Num uniques 1018; 2023-04-05 09:53:44,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:53:44,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688424103, max timestamp=1680688424151Num uniques 1018; 2023-04-05 09:53:44,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:53:44,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688424103, max timestamp=1680688424151Num uniques 1009; 2023-04-05 09:53:44,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-05 09:53:44,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688484265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688484266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688484271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688484274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688484272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688484375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688484379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688484380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688484381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688484480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,489 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:53:44,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688484580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688484587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688484588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688484589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,603 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-05 09:53:44,604 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-05 09:53:44,609 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/53ab3541c95142fbb151419691619773 2023-04-05 09:53:44,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/7cd045e6b18e4189a6edf00123edd056 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056 2023-04-05 09:53:44,639 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056, entries=150, sequenceid=58, filesize=11.6 K 2023-04-05 09:53:44,641 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/41a2d84475d3468285bfceeb3c3bd85f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f 2023-04-05 09:53:44,664 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f, entries=150, sequenceid=58, filesize=11.6 K 2023-04-05 09:53:44,671 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/53ab3541c95142fbb151419691619773 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773 2023-04-05 09:53:44,681 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773, entries=150, sequenceid=58, filesize=11.6 K 2023-04-05 09:53:44,683 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6f2b27e5e74b06149aed9138d7651e34 in 1538ms, sequenceid=58, compaction requested=true 2023-04-05 09:53:44,683 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:44,683 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:44,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:44,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:44,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:44,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:44,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:44,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c8d9b3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:44,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:44,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:44,692 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,692 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:44,695 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:44,700 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:44,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:44,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:44,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:44,700 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:44,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,712 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:44,712 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:44,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:44,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:44,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:44,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:44,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:44,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:44,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:44,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:44,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:44,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=140.89 KB heapSize=404.77 KB 2023-04-05 09:53:44,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:44,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 09:53:44,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:44,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 09:53:44,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:44,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 09:53:44,816 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:44,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:44,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:44,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:44,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:44,826 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:44,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:44,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:44,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:44,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:44,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:44,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:44,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:44,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:44,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/4f3fa34a531448898b61c1ebec0a6804 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:44,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:44,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:44,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:44,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,850 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688484848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0ef9d5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:44,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:44,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,859 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:44,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:44,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:44,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:44,868 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:44,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:44,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:44,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:44,868 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:44,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:44,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:44,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,885 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:44,885 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:44,886 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:44,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:44,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:44,885 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:44,885 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:44,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:44,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:44,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:44,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:44,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:44,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688484894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688484894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688484896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688484896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/c62d496dc27f4d57a886b03aeba68bc6 2023-04-05 09:53:44,941 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/1926953e1d514138896ad20342bfe162 2023-04-05 09:53:44,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/4f3fa34a531448898b61c1ebec0a6804 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804 2023-04-05 09:53:44,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:44,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688484956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:44,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804, entries=200, sequenceid=113, filesize=13.9 K 2023-04-05 09:53:44,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/c62d496dc27f4d57a886b03aeba68bc6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6 2023-04-05 09:53:44,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6, entries=150, sequenceid=113, filesize=11.6 K 2023-04-05 09:53:44,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/1926953e1d514138896ad20342bfe162 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162 2023-04-05 09:53:44,986 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:44,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:45,010 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:45,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:45,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:45,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:45,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162, entries=150, sequenceid=113, filesize=11.6 K 2023-04-05 09:53:45,018 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,018 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~404.72 KB/414432, currentSize=40.25 KB/41220 for 6f2b27e5e74b06149aed9138d7651e34 in 232ms, sequenceid=113, compaction requested=true 2023-04-05 09:53:45,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:45,020 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:45,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:45,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:45,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:45,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:45,041 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,041 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:45,043 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 09:53:45,043 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:45,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:45,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:45,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:53:45,045 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:53:45,045 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:53:45,052 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:53:45,054 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:45,054 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,054 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=46.4 K 2023-04-05 09:53:45,056 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 55c18415f39546b08dc3bf5884f77cb2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680688420881 2023-04-05 09:53:45,057 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c656266c7239417bbc8aaebe9af6cde6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688421159 2023-04-05 09:53:45,059 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 41a2d84475d3468285bfceeb3c3bd85f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680688421896 2023-04-05 09:53:45,060 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c62d496dc27f4d57a886b03aeba68bc6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424269 2023-04-05 09:53:45,061 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:53:45,062 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:45,062 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,062 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=51 K 2023-04-05 09:53:45,064 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 32794aceae3c4bac85cf05ff2bfd3984, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680688420881 2023-04-05 09:53:45,064 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1d8546bd0fc84443a6eff937dfc8c929, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688421159 2023-04-05 09:53:45,068 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7cd045e6b18e4189a6edf00123edd056, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680688421896 2023-04-05 09:53:45,069 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4f3fa34a531448898b61c1ebec0a6804, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424261 2023-04-05 09:53:45,102 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#13 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:45,104 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/c074a3b9dcee4ac588555741758ef450 2023-04-05 09:53:45,112 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#14 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:45,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:45,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:45,173 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/91260d6bef2c44ea97f2a655b598ddeb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91260d6bef2c44ea97f2a655b598ddeb 2023-04-05 09:53:45,214 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 91260d6bef2c44ea97f2a655b598ddeb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:45,214 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:45,214 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=12, startTime=1680688425045; duration=0sec 2023-04-05 09:53:45,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:45,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:53:45,222 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:53:45,222 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:45,222 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,222 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=46.4 K 2023-04-05 09:53:45,223 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d3e50ab59b0841478c7f59487002522e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680688420881 2023-04-05 09:53:45,224 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 17f05006c4744dd5bd4b1db1511dfc15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688421159 2023-04-05 09:53:45,232 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 53ab3541c95142fbb151419691619773, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680688421896 2023-04-05 09:53:45,233 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1926953e1d514138896ad20342bfe162, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424269 2023-04-05 09:53:45,255 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#16 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:45,302 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/149531d078e54c838b93bdd7f2cdeabe as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/149531d078e54c838b93bdd7f2cdeabe 2023-04-05 09:53:45,316 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 149531d078e54c838b93bdd7f2cdeabe(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:45,317 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:45,317 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=12, startTime=1680688425045; duration=0sec 2023-04-05 09:53:45,317 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:45,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688425264, max timestamp=1680688425318Num uniques 859; 2023-04-05 09:53:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688425264, max timestamp=1680688425318Num uniques 859; 2023-04-05 09:53:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688425264, max timestamp=1680688425318Num uniques 859; 2023-04-05 09:53:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:45,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688485401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688485401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688485402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688485404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688485407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688485508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688485508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,603 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/30d4e80ae1824bd0b55c79a1f4e4a77a 2023-04-05 09:53:45,614 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/db6e4b0eed7048eaa5ebbd97e722a9e1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/db6e4b0eed7048eaa5ebbd97e722a9e1 2023-04-05 09:53:45,646 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into db6e4b0eed7048eaa5ebbd97e722a9e1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:45,647 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:45,647 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=12, startTime=1680688425029; duration=0sec 2023-04-05 09:53:45,647 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:45,687 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/2103772c543c48e2b9870408cd5a391c 2023-04-05 09:53:45,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/c074a3b9dcee4ac588555741758ef450 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450 2023-04-05 09:53:45,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688485712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:45,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688485713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,725 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450, entries=150, sequenceid=122, filesize=11.6 K 2023-04-05 09:53:45,729 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/30d4e80ae1824bd0b55c79a1f4e4a77a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a 2023-04-05 09:53:45,750 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a, entries=150, sequenceid=122, filesize=11.6 K 2023-04-05 09:53:45,760 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/2103772c543c48e2b9870408cd5a391c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c 2023-04-05 09:53:45,770 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c, entries=150, sequenceid=122, filesize=11.6 K 2023-04-05 09:53:45,775 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 6f2b27e5e74b06149aed9138d7651e34 in 734ms, sequenceid=122, compaction requested=false 2023-04-05 09:53:45,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:45,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:45,779 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:45,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ffe6385[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:45,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:45,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:45,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:45,786 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:45,791 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:45,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:45,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:45,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:45,792 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:45,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:45,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:45,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:45,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:45,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:45,800 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:45,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:45,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:45,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:45,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:45,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:45,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:45,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:45,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:45,868 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 09:53:45,869 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-05 09:53:45,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:45,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:45,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:45,910 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:45,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:45,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:45,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:45,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,914 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:45,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:45,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:45,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:45,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:45,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:45,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:45,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:45,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:45,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:45,923 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:45,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:46,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688486035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688486038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688486140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688486142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,344 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/aa5a2c8ce3c44545ba4057f56afeef75 2023-04-05 09:53:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688486349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688486350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,373 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/5689fa08d61d462098cc70bdc170b660 2023-04-05 09:53:46,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688486433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,434 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/2e0ecc25d3cd480c94ee081683e9561b 2023-04-05 09:53:46,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688486433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688486433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,447 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/aa5a2c8ce3c44545ba4057f56afeef75 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75 2023-04-05 09:53:46,455 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75, entries=150, sequenceid=162, filesize=11.8 K 2023-04-05 09:53:46,460 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/5689fa08d61d462098cc70bdc170b660 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660 2023-04-05 09:53:46,469 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660, entries=150, sequenceid=162, filesize=11.8 K 2023-04-05 09:53:46,472 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/2e0ecc25d3cd480c94ee081683e9561b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b 2023-04-05 09:53:46,487 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b, entries=150, sequenceid=162, filesize=11.8 K 2023-04-05 09:53:46,492 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 6f2b27e5e74b06149aed9138d7651e34 in 565ms, sequenceid=162, compaction requested=true 2023-04-05 09:53:46,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:46,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,494 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:46,494 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dcb4de8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:46,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:46,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:46,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:46,511 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:46,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:46,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:46,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:46,512 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:46,514 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,514 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,515 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:46,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:46,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,521 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,521 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:46,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,521 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:46,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:46,621 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:46,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:46,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:46,627 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:46,627 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:46,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:46,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:46,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:46,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:46,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:46,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:46,634 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:46,634 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:46,635 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 09:53:46,635 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:46,635 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:46,635 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:46,635 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:46,635 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:46,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:46,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:46,677 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/42fa98fdf3084da9bc2eb1137097e193 2023-04-05 09:53:46,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688426690, max timestamp=1680688426703Num uniques 859; 2023-04-05 09:53:46,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:46,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688426690, max timestamp=1680688426703Num uniques 859; 2023-04-05 09:53:46,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688426690, max timestamp=1680688426703Num uniques 859; 2023-04-05 09:53:46,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:46,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:46,713 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/aabc24bee00d476fa6bc859c2a3ea82d 2023-04-05 09:53:46,757 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/ad7a1824718c48f2bb71285ca4450cd5 2023-04-05 09:53:46,767 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/42fa98fdf3084da9bc2eb1137097e193 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193 2023-04-05 09:53:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33036 deadline: 1680688486763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33042 deadline: 1680688486767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,777 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193, entries=150, sequenceid=171, filesize=11.8 K 2023-04-05 09:53:46,782 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/aabc24bee00d476fa6bc859c2a3ea82d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d 2023-04-05 09:53:46,790 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d, entries=150, sequenceid=171, filesize=11.8 K 2023-04-05 09:53:46,795 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/ad7a1824718c48f2bb71285ca4450cd5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5 2023-04-05 09:53:46,802 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5, entries=150, sequenceid=171, filesize=11.8 K 2023-04-05 09:53:46,804 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 6f2b27e5e74b06149aed9138d7651e34 in 169ms, sequenceid=171, compaction requested=true 2023-04-05 09:53:46,805 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:46,805 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:46,805 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:46,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:46,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a6918a3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:46,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:46,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:46,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:46,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:46,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:46,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:46,823 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:46,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:46,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:46,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:46,824 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:46,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:46,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:46,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:46,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:46,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:46,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:46,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:46,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:46,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688426754, max timestamp=1680688426871Num uniques 1068; 2023-04-05 09:53:46,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:53:46,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=107.34 KB heapSize=302.36 KB 2023-04-05 09:53:46,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:46,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688426754, max timestamp=1680688426871Num uniques 1068; 2023-04-05 09:53:46,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:53:46,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688426754, max timestamp=1680688426871Num uniques 1068; 2023-04-05 09:53:46,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:53:46,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:46,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:46,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5ade36be4cc94770ae22dbc195d6641a 2023-04-05 09:53:46,934 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:46,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:46,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688426910, max timestamp=1680688426942Num uniques 859; 2023-04-05 09:53:46,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688426910, max timestamp=1680688426942Num uniques 859; 2023-04-05 09:53:46,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:53:46,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:53:46,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688426910, max timestamp=1680688426942Num uniques 859; 2023-04-05 09:53:46,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:53:46,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:46,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:46,946 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:46,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:46,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:46,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:46,948 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,948 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:46,950 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:46,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:46,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:46,956 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:46,957 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:46,957 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:46,957 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:46,957 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:46,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:46,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:46,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:46,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:46,966 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,966 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10138c6a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:46,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:46,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:46,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:46,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:46,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:46,979 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:46,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:46,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:46,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:46,979 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:46,982 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:46,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:46,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:46,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:46,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:46,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:46,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:46,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:46,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:46,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:46,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:46,993 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:46,993 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:46,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:46,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:46,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:47,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680688426994, max timestamp=1680688427016Num uniques 1118; 2023-04-05 09:53:47,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-05 09:53:47,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680688426994, max timestamp=1680688427016Num uniques 1118; 2023-04-05 09:53:47,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-05 09:53:47,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680688426994, max timestamp=1680688427016Num uniques 1118; 2023-04-05 09:53:47,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-05 09:53:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680688427060, max timestamp=1680688427079Num uniques 1077; 2023-04-05 09:53:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680688427060, max timestamp=1680688427079Num uniques 1077; 2023-04-05 09:53:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-05 09:53:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-05 09:53:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680688427060, max timestamp=1680688427079Num uniques 1077; 2023-04-05 09:53:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-05 09:53:47,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,104 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,110 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,117 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,117 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,117 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,117 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:47,120 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c6d37a6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:47,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:47,125 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:47,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:47,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:47,126 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,126 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:47,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:47,135 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:47,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:47,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:47,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:47,135 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:47,137 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,137 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,137 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:47,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:47,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,149 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,149 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,149 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:47,149 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:47,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688427118, max timestamp=1680688427179Num uniques 1136; 2023-04-05 09:53:47,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-05 09:53:47,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688427118, max timestamp=1680688427179Num uniques 1136; 2023-04-05 09:53:47,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-05 09:53:47,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688427118, max timestamp=1680688427179Num uniques 1136; 2023-04-05 09:53:47,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-05 09:53:47,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680688427214, max timestamp=1680688427242Num uniques 1145; 2023-04-05 09:53:47,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-05 09:53:47,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680688427214, max timestamp=1680688427242Num uniques 1145; 2023-04-05 09:53:47,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-05 09:53:47,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680688427214, max timestamp=1680688427242Num uniques 1145; 2023-04-05 09:53:47,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-05 09:53:47,249 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,272 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,272 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:47,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1e9fce[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:47,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:47,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:47,293 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:47,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:47,303 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:47,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:47,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:47,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:47,303 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:47,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:47,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:47,314 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:47,314 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:47,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688487324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688487328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/210a3986527c4d64a3ab9535e51300f1 2023-04-05 09:53:47,415 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,420 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,423 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,426 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,426 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,426 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,426 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,428 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d0280e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:47,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:47,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688487430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688487432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:47,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:47,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:47,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:47,444 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:47,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:47,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:47,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:47,445 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:47,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:47,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,454 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:47,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:47,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:47,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,559 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,562 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:47,568 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ac9382[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:47,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:47,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,572 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,573 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:47,573 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:47,573 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:47,573 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,573 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:47,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:47,579 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:47,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:47,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:47,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:47,579 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:47,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,583 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:47,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:47,588 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:47,588 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:47,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688487635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688487637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,688 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,693 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,696 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,699 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,699 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:47,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50195296[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:47,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:47,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:47,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:47,709 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:47,714 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:47,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:47,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:47,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:47,714 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:47,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,723 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:47,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:47,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:47,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:47,723 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,723 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:47,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:47,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/6a0d4ff72d944ba0a08ec499c1343d40 2023-04-05 09:53:47,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5ade36be4cc94770ae22dbc195d6641a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a 2023-04-05 09:53:47,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a, entries=200, sequenceid=209, filesize=14.1 K 2023-04-05 09:53:47,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/210a3986527c4d64a3ab9535e51300f1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1 2023-04-05 09:53:47,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1, entries=150, sequenceid=209, filesize=11.8 K 2023-04-05 09:53:47,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/6a0d4ff72d944ba0a08ec499c1343d40 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40 2023-04-05 09:53:47,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40, entries=150, sequenceid=209, filesize=11.8 K 2023-04-05 09:53:47,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=161.02 KB/164880 for 6f2b27e5e74b06149aed9138d7651e34 in 947ms, sequenceid=209, compaction requested=true 2023-04-05 09:53:47,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:47,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:47,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:47,821 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:53:47,821 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:53:47,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:47,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:47,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:47,827 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:53:47,827 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60018 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:53:47,828 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:47,828 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:47,828 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,828 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,828 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/db6e4b0eed7048eaa5ebbd97e722a9e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=60.9 K 2023-04-05 09:53:47,828 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91260d6bef2c44ea97f2a655b598ddeb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=58.6 K 2023-04-05 09:53:47,829 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting db6e4b0eed7048eaa5ebbd97e722a9e1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424269 2023-04-05 09:53:47,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:47,830 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 91260d6bef2c44ea97f2a655b598ddeb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424269 2023-04-05 09:53:47,830 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c074a3b9dcee4ac588555741758ef450, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680688424821 2023-04-05 09:53:47,831 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 30d4e80ae1824bd0b55c79a1f4e4a77a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680688424821 2023-04-05 09:53:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:47,831 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting aa5a2c8ce3c44545ba4057f56afeef75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680688425391 2023-04-05 09:53:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:47,831 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5689fa08d61d462098cc70bdc170b660, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680688425391 2023-04-05 09:53:47,831 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:47,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:47,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:47,834 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 42fa98fdf3084da9bc2eb1137097e193, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680688426029 2023-04-05 09:53:47,834 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting aabc24bee00d476fa6bc859c2a3ea82d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680688426029 2023-04-05 09:53:47,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,835 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5ade36be4cc94770ae22dbc195d6641a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:47,835 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 210a3986527c4d64a3ab9535e51300f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:47,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:47,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:47,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:47,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:47,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:47,845 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:47,852 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,852 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:47,852 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=161.02 KB heapSize=447.23 KB 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:47,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:47,854 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:47,859 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#28 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:47,916 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d0ec78bcf29541fd8237b04a897d0109 2023-04-05 09:53:47,926 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/50b1fba8bf8d4cafbf42ff74be79d6bc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/50b1fba8bf8d4cafbf42ff74be79d6bc 2023-04-05 09:53:47,940 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 50b1fba8bf8d4cafbf42ff74be79d6bc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:47,940 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:47,940 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=11, startTime=1680688427821; duration=0sec 2023-04-05 09:53:47,940 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:47,941 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:53:47,943 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60018 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:53:47,943 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:47,944 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:47,951 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/149531d078e54c838b93bdd7f2cdeabe, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=58.6 K 2023-04-05 09:53:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:47,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:47,954 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 149531d078e54c838b93bdd7f2cdeabe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680688424269 2023-04-05 09:53:47,955 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2103772c543c48e2b9870408cd5a391c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680688424821 2023-04-05 09:53:47,956 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2e0ecc25d3cd480c94ee081683e9561b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680688425391 2023-04-05 09:53:47,957 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ad7a1824718c48f2bb71285ca4450cd5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680688426029 2023-04-05 09:53:47,957 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6a0d4ff72d944ba0a08ec499c1343d40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:47,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688487965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688487971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:47,980 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:48,000 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/2dd1f8fa808e402a8fe7fadf4d618236 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2dd1f8fa808e402a8fe7fadf4d618236 2023-04-05 09:53:48,008 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 2dd1f8fa808e402a8fe7fadf4d618236(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:48,008 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:48,008 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=11, startTime=1680688427821; duration=0sec 2023-04-05 09:53:48,008 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688488072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688488073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688488277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688488277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,296 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/0064ec258ec94477a7ed119fc13edcf4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0064ec258ec94477a7ed119fc13edcf4 2023-04-05 09:53:48,303 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into 0064ec258ec94477a7ed119fc13edcf4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:48,303 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:48,303 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=11, startTime=1680688427821; duration=0sec 2023-04-05 09:53:48,303 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:48,356 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/0843b47d95004e1a8dc80105f017c741 2023-04-05 09:53:48,376 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/57586d1f9ce945d18cbed42064a6eb89 2023-04-05 09:53:48,382 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d0ec78bcf29541fd8237b04a897d0109 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109 2023-04-05 09:53:48,389 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:53:48,391 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/0843b47d95004e1a8dc80105f017c741 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741 2023-04-05 09:53:48,406 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:53:48,410 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/57586d1f9ce945d18cbed42064a6eb89 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89 2023-04-05 09:53:48,418 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:53:48,421 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~447.19 KB/457920, currentSize=26.84 KB/27480 for 6f2b27e5e74b06149aed9138d7651e34 in 569ms, sequenceid=315, compaction requested=false 2023-04-05 09:53:48,421 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:48,421 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:48,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd647b4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:48,433 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:48,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:48,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:48,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:48,436 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,436 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:48,438 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:48,445 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:48,445 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:48,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:48,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:48,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:48,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:48,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:48,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:48,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:48,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,467 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:48,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,467 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:48,467 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:48,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688428471, max timestamp=1680688428498Num uniques 859; 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688428491, max timestamp=1680688428498Num uniques 859; 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688428471, max timestamp=1680688428498Num uniques 859; 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:53:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:48,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5c73c2a8c00c41dba1b011e31e5e8067 2023-04-05 09:53:48,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688488525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688488525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688488526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,567 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:48,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:48,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:48,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:48,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:48,572 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:48,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:48,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:48,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,579 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688488580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:48,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:48,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:48,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:48,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:48,586 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:48,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688488585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,590 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477e9018[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:48,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:48,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:48,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:48,597 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:48,607 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:48,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:48,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:48,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:48,607 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:48,609 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,609 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,609 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,610 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:48,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:48,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,621 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:48,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,623 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,623 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688488632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688488632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688488632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:48,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:48,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:48,727 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:48,727 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,728 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:48,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:48,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,730 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,730 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,731 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,731 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:48,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,738 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169f3bf6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:48,740 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:48,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:48,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,746 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:48,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:48,750 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:48,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:48,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:48,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:48,750 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:48,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,757 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:48,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,760 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:48,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:48,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688488835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688488837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:48,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688488840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,861 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:48,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:48,865 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:48,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:48,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:48,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:48,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:48,869 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,869 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:48,869 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:48,869 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:48,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c085651[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:48,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:48,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:48,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:48,877 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:48,880 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:48,880 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:48,881 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:48,881 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:48,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,884 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:48,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:48,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:48,889 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,889 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:48,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/77034b1b229c4f59a6600c8a4d219774 2023-04-05 09:53:48,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/ad7385292a464ea2aee39665f5738a73 2023-04-05 09:53:48,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5c73c2a8c00c41dba1b011e31e5e8067 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067 2023-04-05 09:53:48,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067, entries=150, sequenceid=329, filesize=11.9 K 2023-04-05 09:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/77034b1b229c4f59a6600c8a4d219774 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774 2023-04-05 09:53:48,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774, entries=150, sequenceid=329, filesize=11.9 K 2023-04-05 09:53:48,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/ad7385292a464ea2aee39665f5738a73 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73 2023-04-05 09:53:48,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73, entries=150, sequenceid=329, filesize=11.9 K 2023-04-05 09:53:48,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6f2b27e5e74b06149aed9138d7651e34 in 526ms, sequenceid=329, compaction requested=true 2023-04-05 09:53:48,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:48,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:48,984 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:48,984 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:48,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:48,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:48,985 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:48,985 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:48,985 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,985 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:48,986 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0064ec258ec94477a7ed119fc13edcf4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=35.9 K 2023-04-05 09:53:48,986 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:48,986 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:48,986 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/50b1fba8bf8d4cafbf42ff74be79d6bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=35.9 K 2023-04-05 09:53:48,986 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 0064ec258ec94477a7ed119fc13edcf4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:48,987 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 50b1fba8bf8d4cafbf42ff74be79d6bc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:48,988 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d0ec78bcf29541fd8237b04a897d0109, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688427313 2023-04-05 09:53:48,989 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0843b47d95004e1a8dc80105f017c741, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688427313 2023-04-05 09:53:48,989 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5c73c2a8c00c41dba1b011e31e5e8067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:48,989 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 77034b1b229c4f59a6600c8a4d219774, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:48,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:48,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:48,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:48,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:48,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:48,995 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:48,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:48,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:48,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:48,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:48,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:48,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:48,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:48,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:48,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,001 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:49,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,003 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:49,004 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,004 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:49,004 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 09:53:49,004 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:49,004 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,004 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:49,005 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,005 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:49,005 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,052 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/01f359f1b8d64b25ace5b05fc048c63a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/01f359f1b8d64b25ace5b05fc048c63a 2023-04-05 09:53:49,054 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d52169a9ab3a4c8c8f559c937ec0ab04 2023-04-05 09:53:49,060 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/51917c255f1c4ea4967a2c8f12e31fd7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/51917c255f1c4ea4967a2c8f12e31fd7 2023-04-05 09:53:49,064 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 01f359f1b8d64b25ace5b05fc048c63a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,064 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,064 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688428984; duration=0sec 2023-04-05 09:53:49,064 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:49,064 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:49,066 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:49,066 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:49,067 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,067 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2dd1f8fa808e402a8fe7fadf4d618236, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=35.9 K 2023-04-05 09:53:49,067 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2dd1f8fa808e402a8fe7fadf4d618236, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680688426754 2023-04-05 09:53:49,068 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 57586d1f9ce945d18cbed42064a6eb89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688427313 2023-04-05 09:53:49,069 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ad7385292a464ea2aee39665f5738a73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:49,077 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into 51917c255f1c4ea4967a2c8f12e31fd7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,081 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,081 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688428984; duration=0sec 2023-04-05 09:53:49,081 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:49,084 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/71b0a177436343fda253252710670137 2023-04-05 09:53:49,089 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:49,090 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:49,104 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/bcc6d8e8bf394be2ac6ec1909b42070c 2023-04-05 09:53:49,108 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/6c3200e809694b439725535ff9bdb329 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6c3200e809694b439725535ff9bdb329 2023-04-05 09:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d52169a9ab3a4c8c8f559c937ec0ab04 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04 2023-04-05 09:53:49,116 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 6c3200e809694b439725535ff9bdb329(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,116 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,116 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688428984; duration=0sec 2023-04-05 09:53:49,116 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:49,123 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04, entries=150, sequenceid=363, filesize=11.9 K 2023-04-05 09:53:49,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/71b0a177436343fda253252710670137 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137 2023-04-05 09:53:49,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688489121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,135 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137, entries=150, sequenceid=363, filesize=11.9 K 2023-04-05 09:53:49,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688489121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,137 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/bcc6d8e8bf394be2ac6ec1909b42070c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c 2023-04-05 09:53:49,143 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c, entries=150, sequenceid=363, filesize=11.9 K 2023-04-05 09:53:49,144 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 6f2b27e5e74b06149aed9138d7651e34 in 140ms, sequenceid=363, compaction requested=false 2023-04-05 09:53:49,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,145 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:49,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:49,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:49,151 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ae5b83[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:49,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:49,174 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:49,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:49,185 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/290ee47eb10d4e93b7568e7a6759f8d1 2023-04-05 09:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:49,196 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:49,196 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:49,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,204 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688489203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688489203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688489205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:49,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,209 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,209 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:49,209 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:49,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688489237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688489240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,309 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:49,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688489313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688489313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688489313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:49,316 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:49,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:49,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:49,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:49,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:49,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:49,327 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:49,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7263fdfe[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:49,333 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:49,333 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,334 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:49,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:49,337 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:49,339 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:49,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:49,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:49,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:49,340 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:49,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,343 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,347 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:49,348 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:49,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688489439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688489443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,448 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:49,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:49,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:49,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:49,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:49,453 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:49,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:49,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:49,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:49,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:49,460 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,460 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:49,460 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:49,460 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,462 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcc38cf[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:49,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:49,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:49,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:49,469 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:49,473 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:49,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:49,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:49,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:49,474 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:49,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,476 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,476 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,493 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,493 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:49,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,493 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,494 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:49,494 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:49,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688489516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688489517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688489519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:49,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:49,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:49,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:49,600 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:49,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:49,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:49,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,604 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:49,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:49,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:49,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:49,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:49,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,615 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53384ff9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:49,618 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:49,618 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:49,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:49,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:49,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:49,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:49,622 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:49,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:49,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:49,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:49,622 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:49,624 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:49,624 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,624 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:49,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,625 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/e5174046de654ece9de955b4d77c1ab0 2023-04-05 09:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:49,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:49,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:49,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:49,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:49,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:49,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:49,633 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:49,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:49,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:49,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/234909140ca94eaa878d1835ee6f39f6 2023-04-05 09:53:49,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/290ee47eb10d4e93b7568e7a6759f8d1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1 2023-04-05 09:53:49,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1, entries=150, sequenceid=379, filesize=11.9 K 2023-04-05 09:53:49,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/e5174046de654ece9de955b4d77c1ab0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0 2023-04-05 09:53:49,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0, entries=150, sequenceid=379, filesize=11.9 K 2023-04-05 09:53:49,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/234909140ca94eaa878d1835ee6f39f6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6 2023-04-05 09:53:49,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6, entries=150, sequenceid=379, filesize=11.9 K 2023-04-05 09:53:49,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 6f2b27e5e74b06149aed9138d7651e34 in 540ms, sequenceid=379, compaction requested=true 2023-04-05 09:53:49,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:49,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:49,685 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:49,685 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:49,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:49,686 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:49,687 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:49,687 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:49,687 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:49,687 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,687 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,687 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/01f359f1b8d64b25ace5b05fc048c63a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.1 K 2023-04-05 09:53:49,687 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/51917c255f1c4ea4967a2c8f12e31fd7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.1 K 2023-04-05 09:53:49,687 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 01f359f1b8d64b25ace5b05fc048c63a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:49,689 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 51917c255f1c4ea4967a2c8f12e31fd7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:49,690 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 71b0a177436343fda253252710670137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680688428523 2023-04-05 09:53:49,690 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d52169a9ab3a4c8c8f559c937ec0ab04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680688428523 2023-04-05 09:53:49,691 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e5174046de654ece9de955b4d77c1ab0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:49,692 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 290ee47eb10d4e93b7568e7a6759f8d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:49,705 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,707 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,733 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:49,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:49,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:49,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:49,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:49,743 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:49,744 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/de5aecbc06644f0988cf8a3460499f3d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/de5aecbc06644f0988cf8a3460499f3d 2023-04-05 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,746 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:49,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:49,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:49,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:49,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:49,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:49,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688429198, max timestamp=1680688429746Num uniques 859; 2023-04-05 09:53:49,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688429198, max timestamp=1680688429746Num uniques 859; 2023-04-05 09:53:49,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:49,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:49,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:49,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:49,752 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,752 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:49,752 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=51.44 KB heapSize=144.25 KB 2023-04-05 09:53:49,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688429198, max timestamp=1680688429746Num uniques 859; 2023-04-05 09:53:49,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:49,756 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into de5aecbc06644f0988cf8a3460499f3d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,756 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,756 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688429685; duration=0sec 2023-04-05 09:53:49,757 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:49,757 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:49,759 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:49,759 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:49,760 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:49,760 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6c3200e809694b439725535ff9bdb329, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.1 K 2023-04-05 09:53:49,760 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6c3200e809694b439725535ff9bdb329, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680688427969 2023-04-05 09:53:49,761 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting bcc6d8e8bf394be2ac6ec1909b42070c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680688428523 2023-04-05 09:53:49,762 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 234909140ca94eaa878d1835ee6f39f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:49,762 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/ea63486b6e3a4bed8285e59b296db461 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ea63486b6e3a4bed8285e59b296db461 2023-04-05 09:53:49,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:49,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:49,778 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into ea63486b6e3a4bed8285e59b296db461(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,778 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,778 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688429685; duration=0sec 2023-04-05 09:53:49,778 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:49,808 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:49,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688429786, max timestamp=1680688429809Num uniques 859; 2023-04-05 09:53:49,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688429786, max timestamp=1680688429809Num uniques 859; 2023-04-05 09:53:49,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:49,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:49,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688429786, max timestamp=1680688429809Num uniques 859; 2023-04-05 09:53:49,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:49,836 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/ce0e01169d2d483c856d6eaaa95631c8 2023-04-05 09:53:49,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688429841, max timestamp=1680688429859Num uniques 1068; 2023-04-05 09:53:49,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:53:49,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688429841, max timestamp=1680688429859Num uniques 1068; 2023-04-05 09:53:49,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:53:49,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688429841, max timestamp=1680688429859Num uniques 1068; 2023-04-05 09:53:49,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:53:49,894 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/b90b2ebdf9744ee7ae856a88754246c2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/b90b2ebdf9744ee7ae856a88754246c2 2023-04-05 09:53:49,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680688429875, max timestamp=1680688429897Num uniques 1227; 2023-04-05 09:53:49,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=17262, suffix heap size=155576,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=377 2023-04-05 09:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688429882, max timestamp=1680688429897Num uniques 1227; 2023-04-05 09:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-05 09:53:49,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688429882, max timestamp=1680688429897Num uniques 1227; 2023-04-05 09:53:49,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-05 09:53:49,908 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/9b55019a284340dcb0b945013b130f24 2023-04-05 09:53:49,912 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into b90b2ebdf9744ee7ae856a88754246c2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:49,912 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:49,913 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688429685; duration=0sec 2023-04-05 09:53:49,913 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:49,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680688429933, max timestamp=1680688429938Num uniques 1036; 2023-04-05 09:53:49,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 09:53:49,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680688429933, max timestamp=1680688429938Num uniques 1236; 2023-04-05 09:53:49,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-05 09:53:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688429933, max timestamp=1680688429938Num uniques 1036; 2023-04-05 09:53:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-05 09:53:49,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680688429968, max timestamp=1680688429979Num uniques 1045; 2023-04-05 09:53:49,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-05 09:53:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680688429968, max timestamp=1680688429979Num uniques 1045; 2023-04-05 09:53:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-05 09:53:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688429968, max timestamp=1680688429979Num uniques 1009; 2023-04-05 09:53:49,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 09:53:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680688430002, max timestamp=1680688430018Num uniques 1154; 2023-04-05 09:53:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-05 09:53:50,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680688430002, max timestamp=1680688430018Num uniques 1154; 2023-04-05 09:53:50,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-05 09:53:50,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680688430002, max timestamp=1680688430020Num uniques 1009; 2023-04-05 09:53:50,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=9340, suffix heap size=132904,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=204 2023-04-05 09:53:50,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680688430033, max timestamp=1680688430048Num uniques 1113; 2023-04-05 09:53:50,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680688430033, max timestamp=1680688430048Num uniques 1063; 2023-04-05 09:53:50,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=18908, suffix heap size=138520,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=413 2023-04-05 09:53:50,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680688430033, max timestamp=1680688430048Num uniques 1063; 2023-04-05 09:53:50,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=18908, suffix heap size=143720,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=413 2023-04-05 09:53:50,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=18908, suffix heap size=138520,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=413 2023-04-05 09:53:50,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688490072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688490073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688490075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688490077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688490077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688490180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688490182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688490186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688490188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33004 deadline: 1680688490188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,353 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/93a8010e80e44ef49b5c61e009c9c2a6 2023-04-05 09:53:50,362 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/ce0e01169d2d483c856d6eaaa95631c8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8 2023-04-05 09:53:50,371 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8, entries=150, sequenceid=402, filesize=11.9 K 2023-04-05 09:53:50,376 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/9b55019a284340dcb0b945013b130f24 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24 2023-04-05 09:53:50,386 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24, entries=150, sequenceid=402, filesize=11.9 K 2023-04-05 09:53:50,387 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/93a8010e80e44ef49b5c61e009c9c2a6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6 2023-04-05 09:53:50,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688490389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688490397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688490399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688490400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,405 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6, entries=150, sequenceid=402, filesize=11.9 K 2023-04-05 09:53:50,408 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=167.72 KB/171750 for 6f2b27e5e74b06149aed9138d7651e34 in 656ms, sequenceid=402, compaction requested=false 2023-04-05 09:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:50,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688430062, max timestamp=1680688430410Num uniques 1272; 2023-04-05 09:53:50,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=58252, new segment data size=10162, suffix heap size=160256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1272, new segment cells count=222 2023-04-05 09:53:50,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688430062, max timestamp=1680688430410Num uniques 1272; 2023-04-05 09:53:50,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688430062, max timestamp=1680688430410Num uniques 1272; 2023-04-05 09:53:50,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=58252, new segment data size=10162, suffix heap size=160256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1272, new segment cells count=222 2023-04-05 09:53:50,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=58252, new segment data size=10162, suffix heap size=160256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1272, new segment cells count=222 2023-04-05 09:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6079645e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:50,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:50,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,423 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:50,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:50,425 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:50,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=53.67 KB heapSize=140.02 KB 2023-04-05 09:53:50,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:50,429 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:50,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:50,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:50,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:50,430 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:50,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:50,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:50,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:50,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:50,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:50,432 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,432 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,441 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:50,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:50,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:50,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:50,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:50,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a18233123e864c48a87172e1a29b48a4 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688430482, max timestamp=1680688430519Num uniques 859; 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688430482, max timestamp=1680688430519Num uniques 859; 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688430482, max timestamp=1680688430519Num uniques 859; 2023-04-05 09:53:50,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:50,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:50,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:50,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:50,564 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:50,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:50,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:50,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,566 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:50,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:50,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:50,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:50,583 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d98f9a2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:50,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:50,589 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,590 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:50,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:50,597 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:50,603 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:50,603 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:50,603 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:50,603 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:50,603 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:50,605 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,605 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,605 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688490616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,619 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:50,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:50,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,620 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:50,620 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:50,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688490702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688490702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688490703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688490706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,720 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:50,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:50,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:50,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:50,726 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688490720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,728 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,728 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,728 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:50,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:50,732 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,732 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:50,732 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:50,732 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,739 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716ea29e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:50,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:50,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:50,743 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,743 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:50,745 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:50,750 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:50,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:50,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:50,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:50,750 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:50,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,757 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:50,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:50,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,762 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:50,762 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:50,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,862 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:50,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:50,867 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:50,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:50,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:50,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:50,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:50,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:50,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:50,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:50,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:50,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:50,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:50,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:50,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,898 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60386ccd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:50,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:50,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:50,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:50,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:50,908 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:50,908 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:50,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/0d0785cd66a04a61a4f4cb5118485790 2023-04-05 09:53:50,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:50,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:50,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:50,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:50,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:50,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:50,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:50,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:50,921 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:50,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:50,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:50,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:50,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688490928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,026 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,027 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,027 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,029 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:51,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:51,032 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,032 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:51,032 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:51,032 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:51,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:51,034 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:51,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b48af00[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:51,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:51,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:51,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:51,039 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:51,041 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:51,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:51,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:51,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:51,042 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:51,044 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,044 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,044 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:51,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:51,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,150 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,154 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,155 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,156 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,156 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:51,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:51,157 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,157 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:51,158 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:51,158 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,159 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,160 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,160 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1571a22[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:51,161 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:51,161 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,165 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,165 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:51,165 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:51,166 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,167 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:51,170 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:51,171 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:51,171 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:51,171 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:51,171 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:51,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,175 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:51,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:51,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:51,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688491206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688491207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688491207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688491218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688491234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,282 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,284 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,285 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,285 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:51,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:51,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:51,286 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:51,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:51,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:51,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52932c4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,292 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:51,293 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:51,293 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:51,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:51,295 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:51,298 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:51,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:51,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:51,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:51,299 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:51,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,302 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:51,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:51,314 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:51,314 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:51,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f265442828104b1b8ea3d4a44ae3a941 2023-04-05 09:53:51,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a18233123e864c48a87172e1a29b48a4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4 2023-04-05 09:53:51,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4, entries=150, sequenceid=548, filesize=11.9 K 2023-04-05 09:53:51,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/0d0785cd66a04a61a4f4cb5118485790 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790 2023-04-05 09:53:51,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790, entries=150, sequenceid=548, filesize=11.9 K 2023-04-05 09:53:51,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f265442828104b1b8ea3d4a44ae3a941 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941 2023-04-05 09:53:51,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941, entries=150, sequenceid=548, filesize=11.9 K 2023-04-05 09:53:51,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~139.97 KB/143328, currentSize=134.18 KB/137400 for 6f2b27e5e74b06149aed9138d7651e34 in 958ms, sequenceid=548, compaction requested=true 2023-04-05 09:53:51,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:51,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:51,388 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:51,388 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:51,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:51,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:51,390 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:51,390 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:51,390 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,390 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:51,390 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:51,390 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,390 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ea63486b6e3a4bed8285e59b296db461, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.2 K 2023-04-05 09:53:51,390 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/de5aecbc06644f0988cf8a3460499f3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.2 K 2023-04-05 09:53:51,391 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ea63486b6e3a4bed8285e59b296db461, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:51,392 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting de5aecbc06644f0988cf8a3460499f3d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:51,392 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ce0e01169d2d483c856d6eaaa95631c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680688429202 2023-04-05 09:53:51,393 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a18233123e864c48a87172e1a29b48a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:51,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9b55019a284340dcb0b945013b130f24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680688429202 2023-04-05 09:53:51,394 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0d0785cd66a04a61a4f4cb5118485790, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:51,414 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,433 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,444 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:51,459 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:51,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:51,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:51,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:51,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:51,469 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,469 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:51,469 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 09:53:51,470 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:51,499 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/8e7a127774de4809b7f7bc66faa3f3be as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8e7a127774de4809b7f7bc66faa3f3be 2023-04-05 09:53:51,508 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into 8e7a127774de4809b7f7bc66faa3f3be(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:51,508 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:51,508 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688431388; duration=0sec 2023-04-05 09:53:51,508 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:51,508 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:51,512 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:51,512 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:51,512 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,512 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/b90b2ebdf9744ee7ae856a88754246c2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.2 K 2023-04-05 09:53:51,513 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b90b2ebdf9744ee7ae856a88754246c2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680688429099 2023-04-05 09:53:51,513 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 93a8010e80e44ef49b5c61e009c9c2a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680688429202 2023-04-05 09:53:51,514 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f265442828104b1b8ea3d4a44ae3a941, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:51,526 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:51,531 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/e30dbc5c358b46b0b506108e6e2e677d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e30dbc5c358b46b0b506108e6e2e677d 2023-04-05 09:53:51,539 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into e30dbc5c358b46b0b506108e6e2e677d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:51,539 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:51,539 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688431388; duration=0sec 2023-04-05 09:53:51,539 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:51,548 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a7dda09bde64449db2ad9a29fa9b36a7 2023-04-05 09:53:51,584 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/3b848345d2cd4adbb8b88b362faa4473 2023-04-05 09:53:51,603 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/bd28acff5dd14b75b13525cfba33bd47 2023-04-05 09:53:51,611 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a7dda09bde64449db2ad9a29fa9b36a7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7 2023-04-05 09:53:51,619 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7, entries=150, sequenceid=585, filesize=11.9 K 2023-04-05 09:53:51,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/3b848345d2cd4adbb8b88b362faa4473 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473 2023-04-05 09:53:51,626 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473, entries=150, sequenceid=585, filesize=11.9 K 2023-04-05 09:53:51,627 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/bd28acff5dd14b75b13525cfba33bd47 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47 2023-04-05 09:53:51,635 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47, entries=150, sequenceid=585, filesize=11.9 K 2023-04-05 09:53:51,636 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 6f2b27e5e74b06149aed9138d7651e34 in 167ms, sequenceid=585, compaction requested=false 2023-04-05 09:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:51,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26d468a8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:51,648 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:51,648 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,650 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:51,650 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:51,650 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:51,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:51,652 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:51,655 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:51,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:51,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:51,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:51,656 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:51,657 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,660 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,664 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:51,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/6c8f79513bde48fc95c4dac0e08beca5 2023-04-05 09:53:51,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,795 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,797 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:51,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/d30bf709365443c3a4613e19e9af01e3 2023-04-05 09:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28eae3dd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:51,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:51,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688431794, max timestamp=1680688431837Num uniques 859; 2023-04-05 09:53:51,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:51,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688431794, max timestamp=1680688431837Num uniques 859; 2023-04-05 09:53:51,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688431794, max timestamp=1680688431837Num uniques 859; 2023-04-05 09:53:51,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:51,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:51,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:51,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:51,861 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:51,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:51,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:51,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:51,861 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:51,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:51,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:51,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:51,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:51,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:51,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:51,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:51,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688491906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:51,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:51,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:51,973 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/c6e58ccd93fe479288c31619a1ac4d40 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c6e58ccd93fe479288c31619a1ac4d40 2023-04-05 09:53:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:51,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:51,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:51,986 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:51,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:51,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:51,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:51,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:51,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:51,986 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into c6e58ccd93fe479288c31619a1ac4d40(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:51,986 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:51,986 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688431388; duration=0sec 2023-04-05 09:53:51,987 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:52,000 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:52,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:52,000 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:52,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:52,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688492013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:52,024 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,024 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:52,024 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:52,024 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:52,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d57ae8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:52,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:52,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,085 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:52,085 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,085 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:52,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:52,149 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:52,149 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:52,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:52,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,171 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,174 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,200 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,201 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:52,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:52,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:52,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:52,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:52,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:52,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:52,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688492215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688492217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688492218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688492221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688492225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/73ea12dbda7745478e776a13144ebfed 2023-04-05 09:53:52,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/6c8f79513bde48fc95c4dac0e08beca5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5 2023-04-05 09:53:52,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5, entries=250, sequenceid=597, filesize=16.7 K 2023-04-05 09:53:52,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/d30bf709365443c3a4613e19e9af01e3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3 2023-04-05 09:53:52,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3, entries=150, sequenceid=597, filesize=11.9 K 2023-04-05 09:53:52,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/73ea12dbda7745478e776a13144ebfed as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed 2023-04-05 09:53:52,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed, entries=150, sequenceid=597, filesize=11.9 K 2023-04-05 09:53:52,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6f2b27e5e74b06149aed9138d7651e34 in 536ms, sequenceid=597, compaction requested=true 2023-04-05 09:53:52,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:52,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:52,293 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:52,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:52,293 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:52,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:52,295 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42052 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:52,295 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:52,295 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,295 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8e7a127774de4809b7f7bc66faa3f3be, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=41.1 K 2023-04-05 09:53:52,295 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:52,295 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:52,295 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,295 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8e7a127774de4809b7f7bc66faa3f3be, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:52,295 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e30dbc5c358b46b0b506108e6e2e677d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.3 K 2023-04-05 09:53:52,296 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a7dda09bde64449db2ad9a29fa9b36a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680688430585 2023-04-05 09:53:52,296 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e30dbc5c358b46b0b506108e6e2e677d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:52,297 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 3b848345d2cd4adbb8b88b362faa4473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680688430585 2023-04-05 09:53:52,297 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6c8f79513bde48fc95c4dac0e08beca5, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431745 2023-04-05 09:53:52,298 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d30bf709365443c3a4613e19e9af01e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431753 2023-04-05 09:53:52,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:52,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:52,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:52,309 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:52,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:52,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:52,311 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,313 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:52,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:52,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:52,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:52,315 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,315 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:52,316 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:52,334 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/272bf05bc702440d9dbe060841100898 2023-04-05 09:53:52,354 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a59694824b8e40ca81bb628306324c81 2023-04-05 09:53:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:52,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:52,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688492558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688492662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,727 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/e4f45a582fb048569061c8d7db517570 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/e4f45a582fb048569061c8d7db517570 2023-04-05 09:53:52,753 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/894c133d0c8e445e99e6535074d0935c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/894c133d0c8e445e99e6535074d0935c 2023-04-05 09:53:52,756 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into e4f45a582fb048569061c8d7db517570(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:52,756 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:52,756 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688432293; duration=0sec 2023-04-05 09:53:52,756 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:52,756 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:52,758 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:52,758 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:52,758 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,758 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c6e58ccd93fe479288c31619a1ac4d40, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.3 K 2023-04-05 09:53:52,759 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c6e58ccd93fe479288c31619a1ac4d40, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680688430405 2023-04-05 09:53:52,760 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting bd28acff5dd14b75b13525cfba33bd47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680688430585 2023-04-05 09:53:52,760 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 73ea12dbda7745478e776a13144ebfed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431753 2023-04-05 09:53:52,761 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 894c133d0c8e445e99e6535074d0935c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:52,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:52,761 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688432293; duration=0sec 2023-04-05 09:53:52,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:52,768 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:52,770 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/62e59c6b2574464fb2c2c82b687f1b7a 2023-04-05 09:53:52,779 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/272bf05bc702440d9dbe060841100898 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898 2023-04-05 09:53:52,788 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:53:52,789 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a59694824b8e40ca81bb628306324c81 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81 2023-04-05 09:53:52,800 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:53:52,801 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/62e59c6b2574464fb2c2c82b687f1b7a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a 2023-04-05 09:53:52,811 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:53:52,812 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 6f2b27e5e74b06149aed9138d7651e34 in 497ms, sequenceid=634, compaction requested=false 2023-04-05 09:53:52,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:52,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,815 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/6a9fead1cdc64842b9fb94af52f546af as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a9fead1cdc64842b9fb94af52f546af 2023-04-05 09:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:52,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d26d504[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:52,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:52,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:52,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:52,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:52,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:52,827 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,831 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 6a9fead1cdc64842b9fb94af52f546af(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:52,831 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:52,831 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688432293; duration=0sec 2023-04-05 09:53:52,831 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:52,832 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:52,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:52,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:52,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:52,832 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:52,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:52,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:52,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:52,839 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:52,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:52,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:52,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:52,843 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:52,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:52,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:52,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,843 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:52,843 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:52,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:52,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:53:52,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:52,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:52,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:52,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:52,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:52,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:52,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a0097043543042df874b8cc8a5677a08 2023-04-05 09:53:52,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:52,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:52,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:52,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:52,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:52,952 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:52,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:52,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:52,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:52,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:52,954 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:52,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:52,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:52,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688432934, max timestamp=1680688432954Num uniques 859; 2023-04-05 09:53:52,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:52,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:52,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688432934, max timestamp=1680688432954Num uniques 859; 2023-04-05 09:53:52,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:52,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688432934, max timestamp=1680688432954Num uniques 859; 2023-04-05 09:53:52,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:52,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:52,988 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,988 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:52,988 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:52,988 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b26720[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,015 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,015 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,019 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,022 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,023 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,027 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688493033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:53,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688493134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,149 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,150 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,150 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,151 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,152 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,152 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,155 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,155 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,159 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,159 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b993e0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,160 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,161 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,163 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,166 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,166 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,167 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,167 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,167 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,277 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,280 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,280 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,290 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d01bfc5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,301 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,301 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,303 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,303 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,303 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:53,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688493339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/5b6180f33db14af29afaa6211d75bb70 2023-04-05 09:53:53,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,417 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,420 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,424 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e4c794[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,441 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:53,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,446 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,447 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,447 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,547 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,551 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da4250f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,579 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,580 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,597 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,597 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:53,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688493642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,702 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,704 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e987c8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,716 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,722 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,722 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,722 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,722 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,722 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,725 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,728 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,729 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,729 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,729 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,729 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:53,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/75405578f25a42c8afdaab9c083d3feb 2023-04-05 09:53:53,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a0097043543042df874b8cc8a5677a08 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08 2023-04-05 09:53:53,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,833 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08, entries=150, sequenceid=648, filesize=11.9 K 2023-04-05 09:53:53,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,838 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,838 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:53,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/5b6180f33db14af29afaa6211d75bb70 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70 2023-04-05 09:53:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4606f547[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:53,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:53,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:53,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:53,845 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:53,849 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:53,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:53,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:53,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:53,849 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:53,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70, entries=150, sequenceid=648, filesize=11.9 K 2023-04-05 09:53:53,852 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:53,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/75405578f25a42c8afdaab9c083d3feb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb 2023-04-05 09:53:53,852 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,852 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:53,852 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:53,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb, entries=150, sequenceid=648, filesize=11.9 K 2023-04-05 09:53:53,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:53,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:53,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:53,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:53,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:53,858 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:53,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:53,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:53,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 6f2b27e5e74b06149aed9138d7651e34 in 990ms, sequenceid=648, compaction requested=true 2023-04-05 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:53,859 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:53,859 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:53,860 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:53,860 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/e4f45a582fb048569061c8d7db517570, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.4 K 2023-04-05 09:53:53,860 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,861 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/894c133d0c8e445e99e6535074d0935c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.4 K 2023-04-05 09:53:53,861 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e4f45a582fb048569061c8d7db517570, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431753 2023-04-05 09:53:53,861 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 894c133d0c8e445e99e6535074d0935c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431753 2023-04-05 09:53:53,861 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 272bf05bc702440d9dbe060841100898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680688431897 2023-04-05 09:53:53,862 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a59694824b8e40ca81bb628306324c81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680688431897 2023-04-05 09:53:53,862 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a0097043543042df874b8cc8a5677a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:53,863 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5b6180f33db14af29afaa6211d75bb70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:53,872 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:53,872 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:53,886 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/41ddeef8eac54948a1cec7fa033579af as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/41ddeef8eac54948a1cec7fa033579af 2023-04-05 09:53:53,896 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into 41ddeef8eac54948a1cec7fa033579af(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:53,896 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:53,896 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688433859; duration=0sec 2023-04-05 09:53:53,896 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:53,896 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:53:53,897 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:53:53,898 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:53,898 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,898 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a9fead1cdc64842b9fb94af52f546af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.4 K 2023-04-05 09:53:53,898 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6a9fead1cdc64842b9fb94af52f546af, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1680688431753 2023-04-05 09:53:53,899 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 62e59c6b2574464fb2c2c82b687f1b7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680688431897 2023-04-05 09:53:53,899 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 75405578f25a42c8afdaab9c083d3feb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:53,915 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#74 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:53,959 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:53,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:53,962 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:53,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:53,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:53,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,966 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:53,966 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:53,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:53,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:53,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:53,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:53,969 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:53,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:54,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:54,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:54,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688494172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688494226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688494227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688494227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688494227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,232 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:54,234 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:54,235 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:54,236 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:54,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688494276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,283 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/04ccb002b1c849b0acfdbed576ce49a2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/04ccb002b1c849b0acfdbed576ce49a2 2023-04-05 09:53:54,289 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 04ccb002b1c849b0acfdbed576ce49a2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:54,289 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:54,289 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688433859; duration=0sec 2023-04-05 09:53:54,289 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:54,328 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/819d7383acae4a0e9c38c90ea4a08f29 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/819d7383acae4a0e9c38c90ea4a08f29 2023-04-05 09:53:54,334 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 819d7383acae4a0e9c38c90ea4a08f29(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:54,334 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:54,334 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688433859; duration=0sec 2023-04-05 09:53:54,334 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:54,379 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/53085743908940d8a615e552f563f486 2023-04-05 09:53:54,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688494478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688494781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,798 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/eb8588abe21c42759fb6b105ead87013 2023-04-05 09:53:54,825 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb 2023-04-05 09:53:54,832 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/53085743908940d8a615e552f563f486 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486 2023-04-05 09:53:54,842 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486, entries=150, sequenceid=685, filesize=11.9 K 2023-04-05 09:53:54,843 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/eb8588abe21c42759fb6b105ead87013 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013 2023-04-05 09:53:54,851 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013, entries=150, sequenceid=685, filesize=11.9 K 2023-04-05 09:53:54,852 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb 2023-04-05 09:53:54,857 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb, entries=150, sequenceid=685, filesize=11.9 K 2023-04-05 09:53:54,859 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 6f2b27e5e74b06149aed9138d7651e34 in 889ms, sequenceid=685, compaction requested=false 2023-04-05 09:53:54,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:54,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:54,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168e5274[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:54,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:54,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:54,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:54,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:54,870 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:54,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:54,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:54,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:54,870 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:54,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:54,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:54,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:54,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:54,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:54,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:54,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:54,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:54,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:54,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:54,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,883 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:54,883 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:54,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:54,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:54,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:54,988 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:54,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:54,990 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:54,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:54,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:54,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:54,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:54,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:54,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:54,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:54,993 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:54,993 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:55,014 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a8b71a8e20af44efb12ccfe6d8787ea9 2023-04-05 09:53:55,048 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/00de8f37660d4cbab68cb4df95c97e54 2023-04-05 09:53:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:55,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:55,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688435311, max timestamp=1680688435324Num uniques 859; 2023-04-05 09:53:55,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:55,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688435311, max timestamp=1680688435324Num uniques 859; 2023-04-05 09:53:55,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688435311, max timestamp=1680688435324Num uniques 859; 2023-04-05 09:53:55,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:55,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688495355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688495458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,472 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/7009e8e4c156488ab9343eb060a1e724 2023-04-05 09:53:55,478 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/a8b71a8e20af44efb12ccfe6d8787ea9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9 2023-04-05 09:53:55,483 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9, entries=100, sequenceid=698, filesize=9.5 K 2023-04-05 09:53:55,484 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/00de8f37660d4cbab68cb4df95c97e54 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54 2023-04-05 09:53:55,490 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54, entries=100, sequenceid=698, filesize=9.5 K 2023-04-05 09:53:55,491 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/7009e8e4c156488ab9343eb060a1e724 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724 2023-04-05 09:53:55,502 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724, entries=100, sequenceid=698, filesize=9.5 K 2023-04-05 09:53:55,508 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6f2b27e5e74b06149aed9138d7651e34 in 515ms, sequenceid=698, compaction requested=true 2023-04-05 09:53:55,508 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:55,508 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:55,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,510 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9e91f7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:55,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:55,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,515 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:55,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:55,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:55,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:55,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:55,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:55,521 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:55,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:55,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:55,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:55,521 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:55,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:55,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,526 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,532 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:55,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:55,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:55,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:55,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,532 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:55,533 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,533 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:55,533 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:55,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:55,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:55,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:55,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:55,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:55,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:55,635 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:55,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:55,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:55,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:55,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:55,637 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:55,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:55,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:55,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:55,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:55,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:55,639 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:55,639 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:55,640 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:55,640 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:55,650 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/093565d1707c4dad807e5ec8df826931 2023-04-05 09:53:55,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:55,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:55,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688495687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688495792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:55,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688495997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,066 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/ec52cfa900c448f7934f440e90cc52e3 2023-04-05 09:53:56,090 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/69d86ac88883400d988140352bc96459 2023-04-05 09:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/093565d1707c4dad807e5ec8df826931 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931 2023-04-05 09:53:56,104 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931, entries=150, sequenceid=733, filesize=11.9 K 2023-04-05 09:53:56,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/ec52cfa900c448f7934f440e90cc52e3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3 2023-04-05 09:53:56,113 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3, entries=150, sequenceid=733, filesize=11.9 K 2023-04-05 09:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/69d86ac88883400d988140352bc96459 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459 2023-04-05 09:53:56,121 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459, entries=150, sequenceid=733, filesize=11.9 K 2023-04-05 09:53:56,122 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 6f2b27e5e74b06149aed9138d7651e34 in 483ms, sequenceid=733, compaction requested=true 2023-04-05 09:53:56,122 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:56,122 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:56,129 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:56,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d380ab[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:56,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:56,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,133 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:56,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:56,136 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:56,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:56,141 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:56,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:56,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:56,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:56,141 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:56,142 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,142 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:56,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,142 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:56,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:56,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,147 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:56,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:56,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:56,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:56,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,147 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:56,147 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:56,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:56,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:56,248 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:56,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:56,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:56,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:56,256 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:56,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:56,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:56,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:56,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:56,261 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:56,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:56,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:56,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:56,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:56,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:56,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:56,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:56,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:56,266 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:56,266 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:56,266 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:56,269 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:56,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:56,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:56,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688436330, max timestamp=1680688436370Num uniques 859; 2023-04-05 09:53:56,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:56,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688436330, max timestamp=1680688436370Num uniques 859; 2023-04-05 09:53:56,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:56,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688436330, max timestamp=1680688436370Num uniques 859; 2023-04-05 09:53:56,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:53:56,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:56,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688496420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688496524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:56,692 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/0358f174b635461f92cd8181d9f9b769 2023-04-05 09:53:56,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688496730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688497032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,127 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/91c8d216ffd7400e917a7575b33bcdf2 2023-04-05 09:53:57,162 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/69f3beecfea845af8694b23213ab21e2 2023-04-05 09:53:57,174 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/0358f174b635461f92cd8181d9f9b769 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769 2023-04-05 09:53:57,182 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769, entries=150, sequenceid=744, filesize=11.9 K 2023-04-05 09:53:57,183 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/91c8d216ffd7400e917a7575b33bcdf2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2 2023-04-05 09:53:57,193 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2, entries=150, sequenceid=744, filesize=11.9 K 2023-04-05 09:53:57,194 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/69f3beecfea845af8694b23213ab21e2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2 2023-04-05 09:53:57,199 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2, entries=150, sequenceid=744, filesize=11.9 K 2023-04-05 09:53:57,199 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6f2b27e5e74b06149aed9138d7651e34 in 933ms, sequenceid=744, compaction requested=true 2023-04-05 09:53:57,199 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:57,200 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:57,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618ef33b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:57,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:57,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:57,217 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,217 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:57,218 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:57,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:57,221 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:57,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:57,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:57,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:57,222 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:57,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:57,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:57,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:57,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:57,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:57,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:57,338 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:57,338 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,338 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:57,338 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:57,339 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,339 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,340 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:57,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:57,341 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:57,342 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:57,351 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/4ed39d123e4740b9bbd65bcec8c8b22e 2023-04-05 09:53:57,365 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/6d2142cb22a342d6a02c26d684a9266c 2023-04-05 09:53:57,382 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f7cd72af8745445cbf94f571d4df2cee 2023-04-05 09:53:57,387 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/4ed39d123e4740b9bbd65bcec8c8b22e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e 2023-04-05 09:53:57,392 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e, entries=150, sequenceid=780, filesize=11.9 K 2023-04-05 09:53:57,394 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/6d2142cb22a342d6a02c26d684a9266c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c 2023-04-05 09:53:57,420 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c, entries=150, sequenceid=780, filesize=11.9 K 2023-04-05 09:53:57,421 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f7cd72af8745445cbf94f571d4df2cee as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee 2023-04-05 09:53:57,426 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee, entries=150, sequenceid=780, filesize=11.9 K 2023-04-05 09:53:57,427 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 6f2b27e5e74b06149aed9138d7651e34 in 85ms, sequenceid=780, compaction requested=true 2023-04-05 09:53:57,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:57,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:57,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:57,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@577108c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:57,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,433 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,433 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:57,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:57,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:57,441 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:57,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:57,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:57,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:57,442 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:57,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:57,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,450 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:57,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:57,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:57,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:57,554 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:57,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:57,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:57,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:57,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:57,590 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,590 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:57,590 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:57,590 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688437581, max timestamp=1680688437621Num uniques 859; 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688437581, max timestamp=1680688437621Num uniques 859; 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688437581, max timestamp=1680688437621Num uniques 859; 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:57,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:57,626 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9838fc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:57,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:57,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:57,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:57,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:57,667 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:57,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:57,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:57,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:57,667 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:57,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:57,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,680 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:57,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:57,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688497690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,781 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:57,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:57,784 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:57,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:57,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:57,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,786 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:57,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:57,788 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,788 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:57,788 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:57,788 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:57,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@774239fc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:57,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:57,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,793 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:57,793 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,793 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:57,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:57,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688497797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:57,802 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:57,802 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:57,802 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:57,802 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:57,803 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:57,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,805 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,806 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,808 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:57,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:57,810 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:57,810 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:57,910 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:57,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:57,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:57,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:57,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:57,914 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:57,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:57,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:57,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,916 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,916 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:57,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:57,918 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,918 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:57,918 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:57,918 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:57,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:57,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:57,920 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,920 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,921 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,921 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61430b04[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:57,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:57,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,923 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:57,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:57,928 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:57,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:57,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:57,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:57,928 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:57,930 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,930 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:57,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,930 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:57,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,932 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:57,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:57,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:57,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:57,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:57,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:57,935 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:57,935 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:57,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:57,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/44b6391d665843438ccd9269a5a294fc 2023-04-05 09:53:58,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688498001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,038 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,039 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,039 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,040 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,040 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,045 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8d84d0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,048 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,055 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:58,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,164 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66680865[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,173 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,173 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,173 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,175 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,178 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,178 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33022 deadline: 1680688498251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,256 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:58,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33042 deadline: 1680688498257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,258 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:58,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33040 deadline: 1680688498262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,267 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:58,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33036 deadline: 1680688498266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,269 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:53:58,288 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,291 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,293 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83408d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,311 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,311 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,315 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688498309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,316 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,317 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,318 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:58,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/9bc0e29dd6b34e1dbcb80b457556d63a 2023-04-05 09:53:58,418 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,421 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,425 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e160b2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,431 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,435 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,436 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,436 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,443 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,443 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,443 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,443 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,444 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,444 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,544 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,547 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497ad05a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,557 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,562 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,562 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,578 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,578 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,578 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,678 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,682 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,688 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff7c230[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,695 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,695 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,695 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,699 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,699 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,699 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,699 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:53:58,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/8b7c66a99f0c476d9cbdf4d10ab1ad07 2023-04-05 09:53:58,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/44b6391d665843438ccd9269a5a294fc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc 2023-04-05 09:53:58,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc, entries=200, sequenceid=790, filesize=14.3 K 2023-04-05 09:53:58,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/9bc0e29dd6b34e1dbcb80b457556d63a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a 2023-04-05 09:53:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,807 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,808 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,808 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:58,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:58,818 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:58,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688498818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a, entries=100, sequenceid=790, filesize=9.5 K 2023-04-05 09:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/8b7c66a99f0c476d9cbdf4d10ab1ad07 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07 2023-04-05 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ea606b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:58,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:58,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:58,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07, entries=100, sequenceid=790, filesize=9.5 K 2023-04-05 09:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,833 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6f2b27e5e74b06149aed9138d7651e34 in 1280ms, sequenceid=790, compaction requested=true 2023-04-05 09:53:58,833 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:58,833 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:58,834 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:53:58,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:58,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:58,835 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:58,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:58,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:58,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:58,835 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:58,837 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86112 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:53:58,837 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:53:58,837 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,837 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/41ddeef8eac54948a1cec7fa033579af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=84.1 K 2023-04-05 09:53:58,838 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 41ddeef8eac54948a1cec7fa033579af, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:58,838 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81232 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:53:58,838 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:53:58,839 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,839 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/04ccb002b1c849b0acfdbed576ce49a2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=79.3 K 2023-04-05 09:53:58,839 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 53085743908940d8a615e552f563f486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1680688432994 2023-04-05 09:53:58,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,839 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a8b71a8e20af44efb12ccfe6d8787ea9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680688434166 2023-04-05 09:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,839 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 04ccb002b1c849b0acfdbed576ce49a2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:58,840 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 093565d1707c4dad807e5ec8df826931, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680688435347 2023-04-05 09:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,841 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 0358f174b635461f92cd8181d9f9b769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680688435675 2023-04-05 09:53:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,841 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting eb8588abe21c42759fb6b105ead87013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1680688432994 2023-04-05 09:53:58,841 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4ed39d123e4740b9bbd65bcec8c8b22e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680688436411 2023-04-05 09:53:58,841 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 00de8f37660d4cbab68cb4df95c97e54, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680688434166 2023-04-05 09:53:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,842 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ec52cfa900c448f7934f440e90cc52e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680688435347 2023-04-05 09:53:58,842 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 44b6391d665843438ccd9269a5a294fc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688437546 2023-04-05 09:53:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,842 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 91c8d216ffd7400e917a7575b33bcdf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680688435675 2023-04-05 09:53:58,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,843 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6d2142cb22a342d6a02c26d684a9266c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680688436411 2023-04-05 09:53:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:58,844 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9bc0e29dd6b34e1dbcb80b457556d63a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688437547 2023-04-05 09:53:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,846 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:58,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:58,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,847 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:58,847 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:58,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:58,861 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#93 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:58,862 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#94 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:58,882 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/fb41bde8e1a24a02a1a53aebd451185a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/fb41bde8e1a24a02a1a53aebd451185a 2023-04-05 09:53:58,883 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/ba8b5a4ef93542f8b69851b8db45b634 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ba8b5a4ef93542f8b69851b8db45b634 2023-04-05 09:53:58,893 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into fb41bde8e1a24a02a1a53aebd451185a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:58,893 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:58,893 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=9, startTime=1680688438833; duration=0sec 2023-04-05 09:53:58,894 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:53:58,894 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:53:58,896 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into ba8b5a4ef93542f8b69851b8db45b634(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:58,897 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:58,897 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=9, startTime=1680688438834; duration=0sec 2023-04-05 09:53:58,897 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:58,899 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81232 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:53:58,899 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:53:58,899 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,899 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/819d7383acae4a0e9c38c90ea4a08f29, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=79.3 K 2023-04-05 09:53:58,900 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 819d7383acae4a0e9c38c90ea4a08f29, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1680688432548 2023-04-05 09:53:58,901 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c4d22cc459dc48bcb4fcfeefb0a4f3bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1680688432994 2023-04-05 09:53:58,901 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7009e8e4c156488ab9343eb060a1e724, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680688434166 2023-04-05 09:53:58,901 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 69d86ac88883400d988140352bc96459, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680688435347 2023-04-05 09:53:58,902 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 69f3beecfea845af8694b23213ab21e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680688435675 2023-04-05 09:53:58,902 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f7cd72af8745445cbf94f571d4df2cee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680688436411 2023-04-05 09:53:58,902 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8b7c66a99f0c476d9cbdf4d10ab1ad07, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688437547 2023-04-05 09:53:58,916 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#95 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:53:58,947 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:58,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:58,950 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:58,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:58,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:58,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:58,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:58,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:58,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:58,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:58,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:58,955 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:58,955 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:53:59,337 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/a6b8c5824cc048078507b32c27c3bbc8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/a6b8c5824cc048078507b32c27c3bbc8 2023-04-05 09:53:59,353 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into a6b8c5824cc048078507b32c27c3bbc8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:53:59,353 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:59,353 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=9, startTime=1680688438834; duration=0sec 2023-04-05 09:53:59,353 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:53:59,370 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5b95ffb39edc48cc8528504ff681d22d 2023-04-05 09:53:59,382 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/41666d1857fd4b6388f36f14abb3fc62 2023-04-05 09:53:59,794 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f64d351f8ed643ceaa340029c8afb4f3 2023-04-05 09:53:59,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/5b95ffb39edc48cc8528504ff681d22d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d 2023-04-05 09:53:59,807 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d, entries=150, sequenceid=828, filesize=11.9 K 2023-04-05 09:53:59,808 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/41666d1857fd4b6388f36f14abb3fc62 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62 2023-04-05 09:53:59,816 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62, entries=150, sequenceid=828, filesize=11.9 K 2023-04-05 09:53:59,817 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/f64d351f8ed643ceaa340029c8afb4f3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3 2023-04-05 09:53:59,822 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3, entries=150, sequenceid=828, filesize=11.9 K 2023-04-05 09:53:59,823 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 6f2b27e5e74b06149aed9138d7651e34 in 868ms, sequenceid=828, compaction requested=false 2023-04-05 09:53:59,823 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:53:59,823 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:59,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc6a94[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:59,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:59,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:59,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:59,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:59,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:59,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:59,834 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:59,834 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:59,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:59,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,838 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:53:59,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:53:59,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:59,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:59,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:59,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:59,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:59,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:59,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:53:59,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:59,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:53:59,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:53:59,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:59,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/c55ebecd4a7a4837b97195ba9e9fccbc 2023-04-05 09:53:59,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/4ea08ff7aedd40a7973f3aa514b81a5e 2023-04-05 09:53:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688439857, max timestamp=1680688439892Num uniques 859; 2023-04-05 09:53:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688439857, max timestamp=1680688439892Num uniques 859; 2023-04-05 09:53:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688439857, max timestamp=1680688439892Num uniques 859; 2023-04-05 09:53:59,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:59,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:53:59,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:53:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:53:59,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:53:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:53:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:53:59,946 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:53:59,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:59,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:53:59,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:53:59,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:53:59,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688499943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:59,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:59,950 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:53:59,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:53:59,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:53:59,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:53:59,952 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:59,952 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:53:59,953 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:53:59,953 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:53:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:53:59,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181b9231[Count = 0] remaining members to acquire global barrier 2023-04-05 09:53:59,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:53:59,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:53:59,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:53:59,958 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:53:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:53:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:53:59,960 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:53:59,963 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:53:59,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:53:59,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:53:59,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:53:59,963 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:53:59,965 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,965 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:59,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,966 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:53:59,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,968 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:53:59,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:53:59,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:53:59,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:53:59,971 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:53:59,972 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:53:59,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:53:59,972 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:53:59,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:53:59,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688500051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,072 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:00,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:00,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:00,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:00,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:00,080 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:00,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:00,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:00,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,082 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,082 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:00,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:00,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:00,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:00,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:00,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:00,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:54:00,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:54:00,090 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:00,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:00,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:00,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:00,097 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492c3540[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:00,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:00,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:00,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:00,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:00,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:00,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:00,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:00,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:00,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:00,110 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:00,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:00,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:00,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:00,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,219 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:00,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:00,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:00,226 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,229 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:00,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:00,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:00,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:00,232 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,232 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:54:00,232 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:54:00,232 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:00,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f63fd36[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:00,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:00,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,240 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:00,240 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,240 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:00,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,242 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:00,242 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,242 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:00,245 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:00,245 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:00,245 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:00,245 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:00,245 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:00,249 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,249 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:00,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:00,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:00,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:00,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:00,259 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688500254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,259 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,259 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/4c01599332f04982949c19b07db37bd6 2023-04-05 09:54:00,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/c55ebecd4a7a4837b97195ba9e9fccbc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc 2023-04-05 09:54:00,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc, entries=150, sequenceid=839, filesize=11.9 K 2023-04-05 09:54:00,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/4ea08ff7aedd40a7973f3aa514b81a5e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e 2023-04-05 09:54:00,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e, entries=100, sequenceid=839, filesize=9.5 K 2023-04-05 09:54:00,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/4c01599332f04982949c19b07db37bd6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6 2023-04-05 09:54:00,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6, entries=100, sequenceid=839, filesize=9.5 K 2023-04-05 09:54:00,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6f2b27e5e74b06149aed9138d7651e34 in 509ms, sequenceid=839, compaction requested=true 2023-04-05 09:54:00,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:00,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:00,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:00,350 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:00,350 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:00,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:54:00,352 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37610 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:00,352 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:54:00,352 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,352 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/fb41bde8e1a24a02a1a53aebd451185a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=36.7 K 2023-04-05 09:54:00,353 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35170 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:00,353 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:54:00,353 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,353 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ba8b5a4ef93542f8b69851b8db45b634, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=34.3 K 2023-04-05 09:54:00,353 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting fb41bde8e1a24a02a1a53aebd451185a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688436419 2023-04-05 09:54:00,353 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ba8b5a4ef93542f8b69851b8db45b634, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688436419 2023-04-05 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5b95ffb39edc48cc8528504ff681d22d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1680688437642 2023-04-05 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 41666d1857fd4b6388f36f14abb3fc62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1680688437642 2023-04-05 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c55ebecd4a7a4837b97195ba9e9fccbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688439837 2023-04-05 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4ea08ff7aedd40a7973f3aa514b81a5e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688439839 2023-04-05 09:54:00,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:00,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:00,380 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:00,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:00,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:00,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:00,384 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:00,384 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:00,389 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:00,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:00,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:00,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. started... 2023-04-05 09:54:00,402 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:54:00,403 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:00,435 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/eb1971d0799e43e99a7201f12792ef32 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/eb1971d0799e43e99a7201f12792ef32 2023-04-05 09:54:00,441 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/9e038541bcee44639cf3183f2c139a3d 2023-04-05 09:54:00,442 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into eb1971d0799e43e99a7201f12792ef32(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:00,442 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:00,442 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688440350; duration=0sec 2023-04-05 09:54:00,442 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:00,442 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:00,445 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35170 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:00,445 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:54:00,445 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,445 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/a6b8c5824cc048078507b32c27c3bbc8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=34.3 K 2023-04-05 09:54:00,446 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a6b8c5824cc048078507b32c27c3bbc8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680688436419 2023-04-05 09:54:00,446 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f64d351f8ed643ceaa340029c8afb4f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1680688437642 2023-04-05 09:54:00,447 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4c01599332f04982949c19b07db37bd6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688439839 2023-04-05 09:54:00,455 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:00,464 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/6389b1cb5a404ecba020443ae3b34bee 2023-04-05 09:54:00,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:00,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. as already flushing 2023-04-05 09:54:00,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688500583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688500687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,817 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:62753 2023-04-05 09:54:00,817 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:00,821 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:62753 2023-04-05 09:54:00,821 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:00,821 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:62753 2023-04-05 09:54:00,821 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:00,822 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:62753 2023-04-05 09:54:00,822 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:00,831 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/193f032a95624de18dbf948b28b8414a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/193f032a95624de18dbf948b28b8414a 2023-04-05 09:54:00,838 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into 193f032a95624de18dbf948b28b8414a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:00,838 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:00,838 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688440350; duration=0sec 2023-04-05 09:54:00,838 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:00,874 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/e4e12b8a29c94a989774c41bd1370643 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/e4e12b8a29c94a989774c41bd1370643 2023-04-05 09:54:00,879 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into e4e12b8a29c94a989774c41bd1370643(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:00,880 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:00,880 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688440350; duration=0sec 2023-04-05 09:54:00,880 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:00,882 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/0713900b24a74a8ebfff27a544186e0b 2023-04-05 09:54:00,887 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/9e038541bcee44639cf3183f2c139a3d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d 2023-04-05 09:54:00,891 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d, entries=150, sequenceid=875, filesize=11.9 K 2023-04-05 09:54:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33004 deadline: 1680688500891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,892 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/6389b1cb5a404ecba020443ae3b34bee as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee 2023-04-05 09:54:00,898 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee, entries=150, sequenceid=875, filesize=11.9 K 2023-04-05 09:54:00,898 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/0713900b24a74a8ebfff27a544186e0b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b 2023-04-05 09:54:00,908 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b, entries=150, sequenceid=875, filesize=11.9 K 2023-04-05 09:54:00,908 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 6f2b27e5e74b06149aed9138d7651e34 in 507ms, sequenceid=875, compaction requested=false 2023-04-05 09:54:00,908 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:00,908 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:00,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b2f9284[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:00,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:00,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,917 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:00,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:00,919 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:00,922 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:00,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:00,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:00,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:00,922 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:00,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:00,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:00,925 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:00,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:00,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:00,928 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:00,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:01,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:01,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:54:01,196 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:62753 2023-04-05 09:54:01,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:54:01,196 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:01,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d519006a47534e91ba34765521acecc6 2023-04-05 09:54:02,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a990e3b96e2e4661bcc4034f8daa5667 2023-04-05 09:54:02,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/c1e200f97e9a478eacdc0590903a8c4c 2023-04-05 09:54:02,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/d519006a47534e91ba34765521acecc6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6 2023-04-05 09:54:02,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6, entries=100, sequenceid=889, filesize=9.5 K 2023-04-05 09:54:02,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a990e3b96e2e4661bcc4034f8daa5667 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667 2023-04-05 09:54:02,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667, entries=100, sequenceid=889, filesize=9.5 K 2023-04-05 09:54:02,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/c1e200f97e9a478eacdc0590903a8c4c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c 2023-04-05 09:54:02,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c, entries=100, sequenceid=889, filesize=9.5 K 2023-04-05 09:54:02,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 6f2b27e5e74b06149aed9138d7651e34 in 1252ms, sequenceid=889, compaction requested=true 2023-04-05 09:54:02,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:02,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:02,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:02,448 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:02,448 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:02,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:02,449 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:02,449 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/A is initiating minor compaction (all files) 2023-04-05 09:54:02,449 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:02,449 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/A in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:02,449 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/B is initiating minor compaction (all files) 2023-04-05 09:54:02,449 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/eb1971d0799e43e99a7201f12792ef32, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=34.4 K 2023-04-05 09:54:02,449 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/B in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:02,450 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/193f032a95624de18dbf948b28b8414a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=34.4 K 2023-04-05 09:54:02,450 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting eb1971d0799e43e99a7201f12792ef32, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688437669 2023-04-05 09:54:02,450 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 193f032a95624de18dbf948b28b8414a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688437669 2023-04-05 09:54:02,450 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9e038541bcee44639cf3183f2c139a3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1680688439932 2023-04-05 09:54:02,450 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6389b1cb5a404ecba020443ae3b34bee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1680688439932 2023-04-05 09:54:02,451 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d519006a47534e91ba34765521acecc6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1680688440580 2023-04-05 09:54:02,451 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a990e3b96e2e4661bcc4034f8daa5667, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1680688440580 2023-04-05 09:54:02,469 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#A#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:02,483 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#B#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:02,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/7e56d6a780a74d20a73c6ae9a8ab692a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7e56d6a780a74d20a73c6ae9a8ab692a 2023-04-05 09:54:02,892 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/de5e93ac933e46dcb3dc1d9b50bcfc2e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/de5e93ac933e46dcb3dc1d9b50bcfc2e 2023-04-05 09:54:02,893 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/A of 6f2b27e5e74b06149aed9138d7651e34 into 7e56d6a780a74d20a73c6ae9a8ab692a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:02,893 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:02,893 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/A, priority=13, startTime=1680688442448; duration=0sec 2023-04-05 09:54:02,893 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:02,893 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:02,894 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:02,895 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 6f2b27e5e74b06149aed9138d7651e34/C is initiating minor compaction (all files) 2023-04-05 09:54:02,895 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2b27e5e74b06149aed9138d7651e34/C in TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:02,895 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/e4e12b8a29c94a989774c41bd1370643, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp, totalSize=34.4 K 2023-04-05 09:54:02,895 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e4e12b8a29c94a989774c41bd1370643, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680688437669 2023-04-05 09:54:02,895 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 0713900b24a74a8ebfff27a544186e0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1680688439932 2023-04-05 09:54:02,896 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c1e200f97e9a478eacdc0590903a8c4c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1680688440580 2023-04-05 09:54:02,897 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/B of 6f2b27e5e74b06149aed9138d7651e34 into de5e93ac933e46dcb3dc1d9b50bcfc2e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:02,898 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:02,898 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/B, priority=13, startTime=1680688442448; duration=0sec 2023-04-05 09:54:02,898 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:02,904 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2b27e5e74b06149aed9138d7651e34#C#compaction#113 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:02,919 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/7b23296f29d24594869c0c0c0a520931 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7b23296f29d24594869c0c0c0a520931 2023-04-05 09:54:02,924 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2b27e5e74b06149aed9138d7651e34/C of 6f2b27e5e74b06149aed9138d7651e34 into 7b23296f29d24594869c0c0c0a520931(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:02,924 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:02,924 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34., storeName=6f2b27e5e74b06149aed9138d7651e34/C, priority=13, startTime=1680688442448; duration=0sec 2023-04-05 09:54:02,924 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:08,276 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:62753 2023-04-05 09:54:08,276 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:08,299 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:62753 2023-04-05 09:54:08,299 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:08,302 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:62753 2023-04-05 09:54:08,302 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:08,319 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:62753 2023-04-05 09:54:08,319 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 349 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4299 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 3927 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1797 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 5391 rows 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1806 2023-04-05 09:54:08,319 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 5418 rows 2023-04-05 09:54:08,320 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:54:08,320 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44ef43e3 to 127.0.0.1:62753 2023-04-05 09:54:08,320 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:08,323 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:54:08,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:54:08,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:08,338 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688448337"}]},"ts":"1680688448337"} 2023-04-05 09:54:08,339 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:54:08,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:08,340 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:54:08,342 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, UNASSIGN}] 2023-04-05 09:54:08,343 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, UNASSIGN 2023-04-05 09:54:08,344 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6f2b27e5e74b06149aed9138d7651e34, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:08,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688448343"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688448343"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688448343"}]},"ts":"1680688448343"} 2023-04-05 09:54:08,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:08,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:08,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6f2b27e5e74b06149aed9138d7651e34, disabling compactions & flushes 2023-04-05 09:54:08,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. after waiting 0 ms 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:08,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6f2b27e5e74b06149aed9138d7651e34 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=A 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=B 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2b27e5e74b06149aed9138d7651e34, store=C 2023-04-05 09:54:08,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:08,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:08,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/8477b04731e7495489daf6e976d9453f 2023-04-05 09:54:08,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:09,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a1f510007a994ee3af584e0b77bc88d1 2023-04-05 09:54:09,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:09,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/375cffd8c7e340d295b6de1771338db9 2023-04-05 09:54:09,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/A/8477b04731e7495489daf6e976d9453f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8477b04731e7495489daf6e976d9453f 2023-04-05 09:54:09,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8477b04731e7495489daf6e976d9453f, entries=100, sequenceid=899, filesize=9.5 K 2023-04-05 09:54:09,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/B/a1f510007a994ee3af584e0b77bc88d1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a1f510007a994ee3af584e0b77bc88d1 2023-04-05 09:54:09,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a1f510007a994ee3af584e0b77bc88d1, entries=100, sequenceid=899, filesize=9.5 K 2023-04-05 09:54:09,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/.tmp/C/375cffd8c7e340d295b6de1771338db9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/375cffd8c7e340d295b6de1771338db9 2023-04-05 09:54:09,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/375cffd8c7e340d295b6de1771338db9, entries=100, sequenceid=899, filesize=9.5 K 2023-04-05 09:54:09,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6f2b27e5e74b06149aed9138d7651e34 in 1250ms, sequenceid=899, compaction requested=false 2023-04-05 09:54:09,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/db6e4b0eed7048eaa5ebbd97e722a9e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0064ec258ec94477a7ed119fc13edcf4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/51917c255f1c4ea4967a2c8f12e31fd7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/de5aecbc06644f0988cf8a3460499f3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8e7a127774de4809b7f7bc66faa3f3be, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/e4f45a582fb048569061c8d7db517570, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/41ddeef8eac54948a1cec7fa033579af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/fb41bde8e1a24a02a1a53aebd451185a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/eb1971d0799e43e99a7201f12792ef32, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6] to archive 2023-04-05 09:54:09,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:09,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/32794aceae3c4bac85cf05ff2bfd3984 2023-04-05 09:54:09,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/1d8546bd0fc84443a6eff937dfc8c929 2023-04-05 09:54:09,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7cd045e6b18e4189a6edf00123edd056 2023-04-05 09:54:09,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4f3fa34a531448898b61c1ebec0a6804 2023-04-05 09:54:09,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/db6e4b0eed7048eaa5ebbd97e722a9e1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/db6e4b0eed7048eaa5ebbd97e722a9e1 2023-04-05 09:54:09,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c074a3b9dcee4ac588555741758ef450 2023-04-05 09:54:09,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/aa5a2c8ce3c44545ba4057f56afeef75 2023-04-05 09:54:09,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/42fa98fdf3084da9bc2eb1137097e193 2023-04-05 09:54:09,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5ade36be4cc94770ae22dbc195d6641a 2023-04-05 09:54:09,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0064ec258ec94477a7ed119fc13edcf4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0064ec258ec94477a7ed119fc13edcf4 2023-04-05 09:54:09,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d0ec78bcf29541fd8237b04a897d0109 2023-04-05 09:54:09,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/51917c255f1c4ea4967a2c8f12e31fd7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/51917c255f1c4ea4967a2c8f12e31fd7 2023-04-05 09:54:09,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5c73c2a8c00c41dba1b011e31e5e8067 2023-04-05 09:54:09,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d52169a9ab3a4c8c8f559c937ec0ab04 2023-04-05 09:54:09,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/de5aecbc06644f0988cf8a3460499f3d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/de5aecbc06644f0988cf8a3460499f3d 2023-04-05 09:54:09,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/290ee47eb10d4e93b7568e7a6759f8d1 2023-04-05 09:54:09,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/ce0e01169d2d483c856d6eaaa95631c8 2023-04-05 09:54:09,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8e7a127774de4809b7f7bc66faa3f3be to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8e7a127774de4809b7f7bc66faa3f3be 2023-04-05 09:54:09,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a18233123e864c48a87172e1a29b48a4 2023-04-05 09:54:09,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a7dda09bde64449db2ad9a29fa9b36a7 2023-04-05 09:54:09,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/6c8f79513bde48fc95c4dac0e08beca5 2023-04-05 09:54:09,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/e4f45a582fb048569061c8d7db517570 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/e4f45a582fb048569061c8d7db517570 2023-04-05 09:54:09,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/272bf05bc702440d9dbe060841100898 2023-04-05 09:54:09,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/41ddeef8eac54948a1cec7fa033579af to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/41ddeef8eac54948a1cec7fa033579af 2023-04-05 09:54:09,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a0097043543042df874b8cc8a5677a08 2023-04-05 09:54:09,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/53085743908940d8a615e552f563f486 2023-04-05 09:54:09,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/a8b71a8e20af44efb12ccfe6d8787ea9 2023-04-05 09:54:09,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/093565d1707c4dad807e5ec8df826931 2023-04-05 09:54:09,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/0358f174b635461f92cd8181d9f9b769 2023-04-05 09:54:09,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/4ed39d123e4740b9bbd65bcec8c8b22e 2023-04-05 09:54:09,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/44b6391d665843438ccd9269a5a294fc 2023-04-05 09:54:09,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/fb41bde8e1a24a02a1a53aebd451185a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/fb41bde8e1a24a02a1a53aebd451185a 2023-04-05 09:54:09,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/5b95ffb39edc48cc8528504ff681d22d 2023-04-05 09:54:09,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/eb1971d0799e43e99a7201f12792ef32 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/eb1971d0799e43e99a7201f12792ef32 2023-04-05 09:54:09,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/c55ebecd4a7a4837b97195ba9e9fccbc 2023-04-05 09:54:09,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/9e038541bcee44639cf3183f2c139a3d 2023-04-05 09:54:09,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/d519006a47534e91ba34765521acecc6 2023-04-05 09:54:09,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91260d6bef2c44ea97f2a655b598ddeb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/50b1fba8bf8d4cafbf42ff74be79d6bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/01f359f1b8d64b25ace5b05fc048c63a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ea63486b6e3a4bed8285e59b296db461, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e30dbc5c358b46b0b506108e6e2e677d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/894c133d0c8e445e99e6535074d0935c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/04ccb002b1c849b0acfdbed576ce49a2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ba8b5a4ef93542f8b69851b8db45b634, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/193f032a95624de18dbf948b28b8414a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667] to archive 2023-04-05 09:54:09,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:09,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/55c18415f39546b08dc3bf5884f77cb2 2023-04-05 09:54:09,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c656266c7239417bbc8aaebe9af6cde6 2023-04-05 09:54:09,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41a2d84475d3468285bfceeb3c3bd85f 2023-04-05 09:54:09,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91260d6bef2c44ea97f2a655b598ddeb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91260d6bef2c44ea97f2a655b598ddeb 2023-04-05 09:54:09,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/c62d496dc27f4d57a886b03aeba68bc6 2023-04-05 09:54:09,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/30d4e80ae1824bd0b55c79a1f4e4a77a 2023-04-05 09:54:09,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5689fa08d61d462098cc70bdc170b660 2023-04-05 09:54:09,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/aabc24bee00d476fa6bc859c2a3ea82d 2023-04-05 09:54:09,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/50b1fba8bf8d4cafbf42ff74be79d6bc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/50b1fba8bf8d4cafbf42ff74be79d6bc 2023-04-05 09:54:09,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/210a3986527c4d64a3ab9535e51300f1 2023-04-05 09:54:09,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0843b47d95004e1a8dc80105f017c741 2023-04-05 09:54:09,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/01f359f1b8d64b25ace5b05fc048c63a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/01f359f1b8d64b25ace5b05fc048c63a 2023-04-05 09:54:09,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/77034b1b229c4f59a6600c8a4d219774 2023-04-05 09:54:09,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/71b0a177436343fda253252710670137 2023-04-05 09:54:09,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ea63486b6e3a4bed8285e59b296db461 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ea63486b6e3a4bed8285e59b296db461 2023-04-05 09:54:09,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e5174046de654ece9de955b4d77c1ab0 2023-04-05 09:54:09,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9b55019a284340dcb0b945013b130f24 2023-04-05 09:54:09,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e30dbc5c358b46b0b506108e6e2e677d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/e30dbc5c358b46b0b506108e6e2e677d 2023-04-05 09:54:09,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/0d0785cd66a04a61a4f4cb5118485790 2023-04-05 09:54:09,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/3b848345d2cd4adbb8b88b362faa4473 2023-04-05 09:54:09,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/894c133d0c8e445e99e6535074d0935c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/894c133d0c8e445e99e6535074d0935c 2023-04-05 09:54:09,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/d30bf709365443c3a4613e19e9af01e3 2023-04-05 09:54:09,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a59694824b8e40ca81bb628306324c81 2023-04-05 09:54:09,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/04ccb002b1c849b0acfdbed576ce49a2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/04ccb002b1c849b0acfdbed576ce49a2 2023-04-05 09:54:09,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/5b6180f33db14af29afaa6211d75bb70 2023-04-05 09:54:09,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/eb8588abe21c42759fb6b105ead87013 2023-04-05 09:54:09,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/00de8f37660d4cbab68cb4df95c97e54 2023-04-05 09:54:09,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ec52cfa900c448f7934f440e90cc52e3 2023-04-05 09:54:09,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/91c8d216ffd7400e917a7575b33bcdf2 2023-04-05 09:54:09,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6d2142cb22a342d6a02c26d684a9266c 2023-04-05 09:54:09,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ba8b5a4ef93542f8b69851b8db45b634 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/ba8b5a4ef93542f8b69851b8db45b634 2023-04-05 09:54:09,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/9bc0e29dd6b34e1dbcb80b457556d63a 2023-04-05 09:54:09,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/41666d1857fd4b6388f36f14abb3fc62 2023-04-05 09:54:09,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/193f032a95624de18dbf948b28b8414a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/193f032a95624de18dbf948b28b8414a 2023-04-05 09:54:09,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/4ea08ff7aedd40a7973f3aa514b81a5e 2023-04-05 09:54:09,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/6389b1cb5a404ecba020443ae3b34bee 2023-04-05 09:54:09,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a990e3b96e2e4661bcc4034f8daa5667 2023-04-05 09:54:09,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/149531d078e54c838b93bdd7f2cdeabe, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2dd1f8fa808e402a8fe7fadf4d618236, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6c3200e809694b439725535ff9bdb329, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/b90b2ebdf9744ee7ae856a88754246c2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c6e58ccd93fe479288c31619a1ac4d40, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a9fead1cdc64842b9fb94af52f546af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/819d7383acae4a0e9c38c90ea4a08f29, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/a6b8c5824cc048078507b32c27c3bbc8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/e4e12b8a29c94a989774c41bd1370643, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c] to archive 2023-04-05 09:54:09,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:09,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/d3e50ab59b0841478c7f59487002522e 2023-04-05 09:54:09,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/17f05006c4744dd5bd4b1db1511dfc15 2023-04-05 09:54:09,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/53ab3541c95142fbb151419691619773 2023-04-05 09:54:09,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/149531d078e54c838b93bdd7f2cdeabe to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/149531d078e54c838b93bdd7f2cdeabe 2023-04-05 09:54:09,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/1926953e1d514138896ad20342bfe162 2023-04-05 09:54:09,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2103772c543c48e2b9870408cd5a391c 2023-04-05 09:54:09,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2e0ecc25d3cd480c94ee081683e9561b 2023-04-05 09:54:09,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7a1824718c48f2bb71285ca4450cd5 2023-04-05 09:54:09,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2dd1f8fa808e402a8fe7fadf4d618236 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/2dd1f8fa808e402a8fe7fadf4d618236 2023-04-05 09:54:09,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a0d4ff72d944ba0a08ec499c1343d40 2023-04-05 09:54:09,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/57586d1f9ce945d18cbed42064a6eb89 2023-04-05 09:54:09,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6c3200e809694b439725535ff9bdb329 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6c3200e809694b439725535ff9bdb329 2023-04-05 09:54:09,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/ad7385292a464ea2aee39665f5738a73 2023-04-05 09:54:09,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bcc6d8e8bf394be2ac6ec1909b42070c 2023-04-05 09:54:09,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/b90b2ebdf9744ee7ae856a88754246c2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/b90b2ebdf9744ee7ae856a88754246c2 2023-04-05 09:54:09,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/234909140ca94eaa878d1835ee6f39f6 2023-04-05 09:54:09,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/93a8010e80e44ef49b5c61e009c9c2a6 2023-04-05 09:54:09,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c6e58ccd93fe479288c31619a1ac4d40 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c6e58ccd93fe479288c31619a1ac4d40 2023-04-05 09:54:09,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f265442828104b1b8ea3d4a44ae3a941 2023-04-05 09:54:09,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/bd28acff5dd14b75b13525cfba33bd47 2023-04-05 09:54:09,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a9fead1cdc64842b9fb94af52f546af to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/6a9fead1cdc64842b9fb94af52f546af 2023-04-05 09:54:09,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/73ea12dbda7745478e776a13144ebfed 2023-04-05 09:54:09,911 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/62e59c6b2574464fb2c2c82b687f1b7a 2023-04-05 09:54:09,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/819d7383acae4a0e9c38c90ea4a08f29 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/819d7383acae4a0e9c38c90ea4a08f29 2023-04-05 09:54:09,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/75405578f25a42c8afdaab9c083d3feb 2023-04-05 09:54:09,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c4d22cc459dc48bcb4fcfeefb0a4f3bb 2023-04-05 09:54:09,915 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7009e8e4c156488ab9343eb060a1e724 2023-04-05 09:54:09,916 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69d86ac88883400d988140352bc96459 2023-04-05 09:54:09,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/69f3beecfea845af8694b23213ab21e2 2023-04-05 09:54:09,918 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f7cd72af8745445cbf94f571d4df2cee 2023-04-05 09:54:09,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/a6b8c5824cc048078507b32c27c3bbc8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/a6b8c5824cc048078507b32c27c3bbc8 2023-04-05 09:54:09,920 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/8b7c66a99f0c476d9cbdf4d10ab1ad07 2023-04-05 09:54:09,921 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/f64d351f8ed643ceaa340029c8afb4f3 2023-04-05 09:54:09,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/e4e12b8a29c94a989774c41bd1370643 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/e4e12b8a29c94a989774c41bd1370643 2023-04-05 09:54:09,923 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/4c01599332f04982949c19b07db37bd6 2023-04-05 09:54:09,924 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/0713900b24a74a8ebfff27a544186e0b 2023-04-05 09:54:09,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/c1e200f97e9a478eacdc0590903a8c4c 2023-04-05 09:54:09,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/recovered.edits/902.seqid, newMaxSeqId=902, maxSeqId=1 2023-04-05 09:54:09,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34. 2023-04-05 09:54:09,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6f2b27e5e74b06149aed9138d7651e34: 2023-04-05 09:54:09,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:09,933 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6f2b27e5e74b06149aed9138d7651e34, regionState=CLOSED 2023-04-05 09:54:09,933 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688449933"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688449933"}]},"ts":"1680688449933"} 2023-04-05 09:54:09,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-05 09:54:09,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 6f2b27e5e74b06149aed9138d7651e34, server=jenkins-hbase4.apache.org,46827,1680688416991 in 1.5890 sec 2023-04-05 09:54:09,937 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-05 09:54:09,937 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2b27e5e74b06149aed9138d7651e34, UNASSIGN in 1.5940 sec 2023-04-05 09:54:09,938 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688449938"}]},"ts":"1680688449938"} 2023-04-05 09:54:09,939 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:54:09,947 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:54:09,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6170 sec 2023-04-05 09:54:10,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 09:54:10,444 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-05 09:54:10,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:54:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,455 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,457 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 09:54:10,462 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:10,465 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/recovered.edits] 2023-04-05 09:54:10,468 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7e56d6a780a74d20a73c6ae9a8ab692a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/7e56d6a780a74d20a73c6ae9a8ab692a 2023-04-05 09:54:10,469 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8477b04731e7495489daf6e976d9453f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/A/8477b04731e7495489daf6e976d9453f 2023-04-05 09:54:10,472 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a1f510007a994ee3af584e0b77bc88d1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/a1f510007a994ee3af584e0b77bc88d1 2023-04-05 09:54:10,473 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/de5e93ac933e46dcb3dc1d9b50bcfc2e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/B/de5e93ac933e46dcb3dc1d9b50bcfc2e 2023-04-05 09:54:10,475 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/375cffd8c7e340d295b6de1771338db9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/375cffd8c7e340d295b6de1771338db9 2023-04-05 09:54:10,476 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7b23296f29d24594869c0c0c0a520931 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/C/7b23296f29d24594869c0c0c0a520931 2023-04-05 09:54:10,479 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/recovered.edits/902.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34/recovered.edits/902.seqid 2023-04-05 09:54:10,480 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/6f2b27e5e74b06149aed9138d7651e34 2023-04-05 09:54:10,480 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:54:10,482 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,504 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:54:10,545 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:54:10,547 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,547 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:54:10,547 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688450547"}]},"ts":"9223372036854775807"} 2023-04-05 09:54:10,549 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:54:10,549 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6f2b27e5e74b06149aed9138d7651e34, NAME => 'TestAcidGuarantees,,1680688420025.6f2b27e5e74b06149aed9138d7651e34.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:54:10,549 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:54:10,549 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688450549"}]},"ts":"9223372036854775807"} 2023-04-05 09:54:10,551 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:54:10,553 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 104 msec 2023-04-05 09:54:10,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 09:54:10,559 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-05 09:54:10,577 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x65923c37-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_544665588_15 at /127.0.0.1:42310 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:46827-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x65923c37-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x65923c37-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x65923c37-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=563 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 332) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1392 (was 2075) 2023-04-05 09:54:10,588 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=169, AvailableMemoryMB=1391 2023-04-05 09:54:10,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:54:10,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:54:10,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:10,593 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:54:10,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-05 09:54:10,594 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:54:10,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:10,595 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:10,596 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 empty. 2023-04-05 09:54:10,596 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:10,596 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:54:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:10,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:11,004 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:54:11,006 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3f59f85b0ea42d0cfdd0e5ffae0c1e23, NAME => 'TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:54:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3f59f85b0ea42d0cfdd0e5ffae0c1e23, disabling compactions & flushes 2023-04-05 09:54:11,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. after waiting 0 ms 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:11,415 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:54:11,416 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688451416"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688451416"}]},"ts":"1680688451416"} 2023-04-05 09:54:11,417 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:54:11,418 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:54:11,418 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688451418"}]},"ts":"1680688451418"} 2023-04-05 09:54:11,419 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:54:11,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, ASSIGN}] 2023-04-05 09:54:11,428 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, ASSIGN 2023-04-05 09:54:11,429 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:54:11,580 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:11,580 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688451580"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688451580"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688451580"}]},"ts":"1680688451580"} 2023-04-05 09:54:11,582 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:11,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:11,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3f59f85b0ea42d0cfdd0e5ffae0c1e23, NAME => 'TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:54:11,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:54:11,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,740 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,741 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A 2023-04-05 09:54:11,741 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A 2023-04-05 09:54:11,741 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:11,742 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName A 2023-04-05 09:54:11,742 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:11,742 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,743 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B 2023-04-05 09:54:11,743 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B 2023-04-05 09:54:11,744 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:11,744 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName B 2023-04-05 09:54:11,744 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:11,744 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,745 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C 2023-04-05 09:54:11,745 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C 2023-04-05 09:54:11,746 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:11,746 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName C 2023-04-05 09:54:11,746 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:11,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:54:11,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:11,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:54:11,752 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3f59f85b0ea42d0cfdd0e5ffae0c1e23; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75445980, jitterRate=0.1242327094078064}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:54:11,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:11,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., pid=18, masterSystemTime=1680688451734 2023-04-05 09:54:11,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,754 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:11,755 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:11,755 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688451755"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688451755"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688451755"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688451755"}]},"ts":"1680688451755"} 2023-04-05 09:54:11,757 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-05 09:54:11,758 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 in 174 msec 2023-04-05 09:54:11,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-05 09:54:11,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, ASSIGN in 331 msec 2023-04-05 09:54:11,760 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:54:11,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688451760"}]},"ts":"1680688451760"} 2023-04-05 09:54:11,761 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:54:11,763 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:54:11,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-05 09:54:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 09:54:12,699 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-05 09:54:12,700 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x31436cf6 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:12,706 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33c943f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:12,707 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:12,712 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:12,713 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:54:12,717 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:54:12,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:54:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:54:12,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:13,147 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:54:13,148 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:54:13,148 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:54:13,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 09:54:13,166 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, REOPEN/MOVE}] 2023-04-05 09:54:13,167 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, REOPEN/MOVE 2023-04-05 09:54:13,168 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,168 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688453168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688453168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688453168"}]},"ts":"1680688453168"} 2023-04-05 09:54:13,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:13,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3f59f85b0ea42d0cfdd0e5ffae0c1e23, disabling compactions & flushes 2023-04-05 09:54:13,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. after waiting 0 ms 2023-04-05 09:54:13,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 09:54:13,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:13,327 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3f59f85b0ea42d0cfdd0e5ffae0c1e23 to self. 2023-04-05 09:54:13,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,329 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=CLOSED 2023-04-05 09:54:13,329 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688453329"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688453329"}]},"ts":"1680688453329"} 2023-04-05 09:54:13,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-05 09:54:13,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 in 161 msec 2023-04-05 09:54:13,332 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=true 2023-04-05 09:54:13,483 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,483 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688453483"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688453483"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688453483"}]},"ts":"1680688453483"} 2023-04-05 09:54:13,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:13,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3f59f85b0ea42d0cfdd0e5ffae0c1e23, NAME => 'TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:54:13,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:54:13,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,643 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,644 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A 2023-04-05 09:54:13,644 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A 2023-04-05 09:54:13,644 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:13,647 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName A 2023-04-05 09:54:13,648 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:13,649 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,650 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B 2023-04-05 09:54:13,650 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B 2023-04-05 09:54:13,650 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:13,651 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName B 2023-04-05 09:54:13,651 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:13,651 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,652 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C 2023-04-05 09:54:13,652 DEBUG [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C 2023-04-05 09:54:13,652 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:13,653 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3f59f85b0ea42d0cfdd0e5ffae0c1e23 columnFamilyName C 2023-04-05 09:54:13,653 INFO [StoreOpener-3f59f85b0ea42d0cfdd0e5ffae0c1e23-1] regionserver.HStore(310): Store=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:13,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:54:13,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3f59f85b0ea42d0cfdd0e5ffae0c1e23; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60997551, jitterRate=-0.0910656601190567}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:54:13,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:13,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., pid=23, masterSystemTime=1680688453637 2023-04-05 09:54:13,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,662 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,662 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688453662"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688453662"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688453662"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688453662"}]},"ts":"1680688453662"} 2023-04-05 09:54:13,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-05 09:54:13,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 in 179 msec 2023-04-05 09:54:13,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-05 09:54:13,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, REOPEN/MOVE in 499 msec 2023-04-05 09:54:13,669 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-05 09:54:13,669 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-04-05 09:54:13,671 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 943 msec 2023-04-05 09:54:13,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-05 09:54:13,680 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x29f78b2c to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,685 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51c62b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,686 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x7bbdba17 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,689 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,689 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,693 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,693 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,699 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,699 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,703 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,703 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,707 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,708 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,714 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,715 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,719 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,719 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:13,724 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:13,729 DEBUG [hconnection-0x7d03c77a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,729 DEBUG [hconnection-0x6305ec71-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,730 DEBUG [hconnection-0x526f2948-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,733 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,733 DEBUG [hconnection-0x27d25b6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,734 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,737 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,738 DEBUG [hconnection-0x35c8190a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,738 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,739 DEBUG [hconnection-0x41e53772-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,740 DEBUG [hconnection-0x62f3bd1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,739 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,740 DEBUG [hconnection-0x5848520c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,741 DEBUG [hconnection-0xd25198b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:13,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:13,743 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,749 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,756 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,757 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:13,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:13,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:13,759 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:13,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:13,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:13,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:13,764 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,764 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,765 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,766 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,766 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:13,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:13,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:13,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:13,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:13,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:13,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:13,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:54:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:13,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:13,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:13,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:13,776 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:13,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:13,794 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:13,794 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,794 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6113ca[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:13,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:13,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:13,801 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:13,801 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,801 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:13,802 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:13,802 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:13,802 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:13,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688513796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688513796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:13,808 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:13,808 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:13,808 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:13,808 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:13,808 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:13,810 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:13,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:13,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688513798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:13,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:13,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:13,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688513799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688513801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:13,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:13,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:13,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,821 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,821 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:13,821 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:13,821 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:13,844 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056ce8f0b08ed74ab6a25de0e47d68bd86 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056ce8f0b08ed74ab6a25de0e47d68bd86 2023-04-05 09:54:13,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688513909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688513909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688513913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688513913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:13,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688513917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,921 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:13,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:13,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:13,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:13,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:13,955 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:13,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:13,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:13,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:13,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:13,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:13,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:13,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:13,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:13,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:13,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:13,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:13,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:13,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:13,984 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,984 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:13,984 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:13,984 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:13,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:13,989 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:13,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:13,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:13,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674fa953[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:13,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:13,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:13,993 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,993 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:13,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:13,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:13,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:13,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:13,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:13,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:13,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,000 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,001 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,002 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,002 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,002 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,004 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,010 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,010 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,116 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688514115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688514115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688514116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688514119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688514121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,180 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,182 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,226 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b572c1[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,235 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,238 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,238 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,240 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,245 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,255 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,255 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,255 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,256 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:14,256 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,320 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f 2023-04-05 09:54:14,356 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,388 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,390 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,394 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,394 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,394 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,394 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,396 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f96901[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,399 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,399 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,400 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,404 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,404 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,407 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,408 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,411 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:14,411 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,411 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,411 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688514420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688514422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688514424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688514425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688514429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,511 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,520 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,522 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,528 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,528 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,528 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,528 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,531 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2741d997[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,534 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,534 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,534 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,538 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,541 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,541 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,545 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,548 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,548 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:14,649 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,657 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,658 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,658 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,660 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,660 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,663 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,663 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,663 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,663 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddee91d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,673 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,673 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,673 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,673 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,673 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,675 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,675 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,675 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,677 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,683 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,684 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,684 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:14,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,784 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,789 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/f3fd3f1dba884b999a5b419efa55167c 2023-04-05 09:54:14,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,804 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,804 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1a71d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,815 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,818 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,819 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,822 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,822 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:14,832 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,833 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:14,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/5119d518e0d24406959000839907f897 2023-04-05 09:54:14,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f 2023-04-05 09:54:14,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f, entries=250, sequenceid=17, filesize=38.7 K 2023-04-05 09:54:14,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/f3fd3f1dba884b999a5b419efa55167c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c 2023-04-05 09:54:14,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 09:54:14,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/5119d518e0d24406959000839907f897 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897 2023-04-05 09:54:14,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 09:54:14,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1124ms, sequenceid=17, compaction requested=false 2023-04-05 09:54:14,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:14,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:14,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:14,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:14,933 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:14,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:14,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688514937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688514938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688514941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688514942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688514941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:14,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:14,956 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:14,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:14,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:14,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,961 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:14,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:14,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:14,965 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052c51af334608473c8412650507be3bef to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052c51af334608473c8412650507be3bef 2023-04-05 09:54:14,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:14,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:14,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:14,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:14,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:14,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2adb6745[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:14,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:14,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:14,977 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:14,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:14,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:14,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:14,982 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:14,982 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:14,982 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:14,982 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:14,982 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:14,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:14,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:14,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:14,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:14,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,997 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:14,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:14,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:14,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:14,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:14,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:14,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:14,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:14,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:14,998 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:14,998 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:15,016 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/681208158d5a42a28d52d860ffc9032b 2023-04-05 09:54:15,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/8a2569048cb2498a8669f2e8579cd9ce 2023-04-05 09:54:15,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688515049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688515049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688515049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688515050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688515054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,098 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:15,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:15,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:15,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:15,104 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,106 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:15,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:15,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:15,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:15,115 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,115 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:15,116 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:15,116 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:15,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f8ee8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:15,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:15,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,122 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:15,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:15,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:15,123 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,123 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:15,125 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:15,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:15,129 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:15,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:15,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:15,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:15,129 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:15,131 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,131 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,131 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:15,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:15,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,138 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:15,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:15,138 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:15,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,239 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:15,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:15,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:15,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688515252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688515252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688515253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688515253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:15,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:15,256 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:15,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:15,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:15,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688515257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,264 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:15,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:15,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:15,280 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,280 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:15,280 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:15,280 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:15,283 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a3199b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:15,286 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:15,286 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:15,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:15,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:15,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:15,296 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:15,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:15,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:15,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:15,296 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:15,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:15,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:15,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,310 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:15,310 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:15,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,410 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:15,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:15,414 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:15,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:15,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:15,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,417 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:15,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:15,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:15,418 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,418 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:15,418 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:15,419 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:15,420 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d6a6fd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:15,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:15,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,424 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:15,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:15,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:15,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:15,425 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:15,429 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:15,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:15,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:15,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:15,429 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:15,431 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,431 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:15,431 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:15,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:15,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,437 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:15,437 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:15,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:15,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/a2e692a5051a41ebb3c78122e0d3fc2e 2023-04-05 09:54:15,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/681208158d5a42a28d52d860ffc9032b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b 2023-04-05 09:54:15,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b, entries=150, sequenceid=37, filesize=25.1 K 2023-04-05 09:54:15,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/8a2569048cb2498a8669f2e8579cd9ce as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce 2023-04-05 09:54:15,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 09:54:15,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/a2e692a5051a41ebb3c78122e0d3fc2e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e 2023-04-05 09:54:15,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 09:54:15,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 579ms, sequenceid=37, compaction requested=false 2023-04-05 09:54:15,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:15,508 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:54:15,537 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:15,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:15,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:15,547 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:15,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:15,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:15,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:15,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:15,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:15,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:15,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:15,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:15,557 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:15,558 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:15,558 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:15,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:15,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:15,596 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054890fd41e993436ea71741f6f6619102 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054890fd41e993436ea71741f6f6619102 2023-04-05 09:54:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688515609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688515611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688515612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688515615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688515615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688515716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688515716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688515717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688515718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688515718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688515919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688515919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688515920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688515921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:15,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688515924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,015 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/e72849dc58bf44acbae498b060f5c149 2023-04-05 09:54:16,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688516222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688516222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688516226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688516226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688516228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,434 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/42de927a7673410fa9ae5fb1b0cc4ba6 2023-04-05 09:54:16,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688516725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688516728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688516729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688516732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:16,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688516734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,845 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/aec64b37aa6344d69f39530964b35b69 2023-04-05 09:54:16,852 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/e72849dc58bf44acbae498b060f5c149 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149 2023-04-05 09:54:16,858 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149, entries=100, sequenceid=49, filesize=18.4 K 2023-04-05 09:54:16,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/42de927a7673410fa9ae5fb1b0cc4ba6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6 2023-04-05 09:54:16,865 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6, entries=100, sequenceid=49, filesize=9.3 K 2023-04-05 09:54:16,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/aec64b37aa6344d69f39530964b35b69 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69 2023-04-05 09:54:16,872 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69, entries=100, sequenceid=49, filesize=9.3 K 2023-04-05 09:54:16,873 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1316ms, sequenceid=49, compaction requested=true 2023-04-05 09:54:16,873 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:16,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:16,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:16,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:16,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c6104d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:16,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:16,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,881 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,881 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:16,881 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:16,881 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:16,881 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,881 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:16,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:16,894 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:16,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:16,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:16,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:16,894 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:16,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:16,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:16,897 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,898 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:16,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:16,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:16,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:16,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:16,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:16,901 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:16,901 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:16,902 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:17,002 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:17,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:17,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:17,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:17,006 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:17,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:17,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:17,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:17,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:17,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:17,008 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:17,008 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:17,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:17,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:17,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:17,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:17,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:17,012 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:17,012 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:17,012 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:17,048 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040543252eede6844bb28d6997ba2c82e19d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040543252eede6844bb28d6997ba2c82e19d 2023-04-05 09:54:17,506 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/4a2fb4cb0dec42599b4a1a1927340d66 2023-04-05 09:54:17,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:17,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:17,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688517749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688517751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688517752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688517753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688517754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688517855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688517855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688517859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688517862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688517862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,942 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/e8f5fee29b2345d4b342e52585140b58 2023-04-05 09:54:17,954 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/fc21a1a307c548588a1f04814f28c760 2023-04-05 09:54:17,959 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/4a2fb4cb0dec42599b4a1a1927340d66 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66 2023-04-05 09:54:17,975 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66, entries=150, sequenceid=69, filesize=25.1 K 2023-04-05 09:54:17,976 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/e8f5fee29b2345d4b342e52585140b58 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58 2023-04-05 09:54:17,986 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58, entries=150, sequenceid=69, filesize=11.6 K 2023-04-05 09:54:17,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/fc21a1a307c548588a1f04814f28c760 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760 2023-04-05 09:54:17,991 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760, entries=150, sequenceid=69, filesize=11.6 K 2023-04-05 09:54:17,992 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 980ms, sequenceid=69, compaction requested=true 2023-04-05 09:54:17,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:17,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:17,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:17,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:17,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:17,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b31d0f3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:17,997 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:17,997 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:17,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:17,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,000 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,002 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,003 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,011 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,011 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:18,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:18,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688518077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688518079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,082 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058421203888cb47c39897867af51820f0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058421203888cb47c39897867af51820f0 2023-04-05 09:54:18,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688518080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688518080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688518080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,088 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/20136159529e4ab6bcf324900aeddb3a 2023-04-05 09:54:18,111 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,121 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,123 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/96efe58dfab14deb8de3a70e32729957 2023-04-05 09:54:18,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,129 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,129 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,131 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0f67cf[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,136 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,139 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,140 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,160 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,160 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:18,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,160 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,160 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,160 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,161 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,160 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,161 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688518181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688518181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688518187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688518187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688518192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,261 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,265 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,267 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9edb3d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,281 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,282 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,282 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,283 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,283 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,283 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,284 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,285 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,287 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,288 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,288 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,288 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:18,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688518385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,388 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688518386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688518388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688518389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,400 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688518395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,401 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,401 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,405 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,414 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,414 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,414 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,417 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6edefa5b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,421 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,423 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,425 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,426 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,428 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,434 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,434 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,434 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,434 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:18,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,435 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,435 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,535 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,539 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,541 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,543 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,543 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,543 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,543 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d193b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,553 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,553 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,553 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,555 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,555 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,555 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,563 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,563 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/79d942576f9649f9adc7eeefa2fbbe8a 2023-04-05 09:54:18,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/20136159529e4ab6bcf324900aeddb3a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a 2023-04-05 09:54:18,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a, entries=250, sequenceid=84, filesize=38.7 K 2023-04-05 09:54:18,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/96efe58dfab14deb8de3a70e32729957 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957 2023-04-05 09:54:18,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 09:54:18,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/79d942576f9649f9adc7eeefa2fbbe8a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a 2023-04-05 09:54:18,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 09:54:18,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 552ms, sequenceid=84, compaction requested=true 2023-04-05 09:54:18,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:18,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:18,614 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:54:18,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:18,614 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:54:18,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:18,616 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 149550 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:54:18,616 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A is initiating minor compaction (all files) 2023-04-05 09:54:18,616 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,616 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=146.0 K 2023-04-05 09:54:18,617 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8acb2090bf3e4bfbbdca2ab3cb9f7e3f, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688453760 2023-04-05 09:54:18,618 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 681208158d5a42a28d52d860ffc9032b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680688453787 2023-04-05 09:54:18,618 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:54:18,618 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B is initiating minor compaction (all files) 2023-04-05 09:54:18,618 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,618 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=55.7 K 2023-04-05 09:54:18,620 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f3fd3f1dba884b999a5b419efa55167c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688453761 2023-04-05 09:54:18,621 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e72849dc58bf44acbae498b060f5c149, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688454937 2023-04-05 09:54:18,623 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8a2569048cb2498a8669f2e8579cd9ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680688453787 2023-04-05 09:54:18,623 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4a2fb4cb0dec42599b4a1a1927340d66, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680688455605 2023-04-05 09:54:18,624 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 42de927a7673410fa9ae5fb1b0cc4ba6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688454937 2023-04-05 09:54:18,624 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 20136159529e4ab6bcf324900aeddb3a, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457751 2023-04-05 09:54:18,624 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e8f5fee29b2345d4b342e52585140b58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680688455605 2023-04-05 09:54:18,625 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 96efe58dfab14deb8de3a70e32729957, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457752 2023-04-05 09:54:18,643 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#B#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:18,664 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,664 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#A#compaction#132 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,681 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,683 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,683 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,685 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:18,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:18,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688518701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688518704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688518704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688518705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688518705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afd228e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,723 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,723 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,727 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,727 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,730 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,730 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,731 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,731 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,731 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,741 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,741 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,751 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040564545ded91584bfdad3251b69989c0b7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564545ded91584bfdad3251b69989c0b7 2023-04-05 09:54:18,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688518806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688518809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688518811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688518811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,841 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,845 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,846 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,846 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,852 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a65fdf[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,856 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,860 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,861 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:18,869 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,869 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,969 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:18,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:18,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:18,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:18,976 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:18,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:18,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:18,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,978 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:18,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:18,980 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,980 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:18,980 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:18,980 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:18,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb37b72[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:18,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:18,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,986 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:18,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:18,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:18,990 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:18,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:18,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:18,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:18,991 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:18,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:18,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:18,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:18,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:18,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:18,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:18,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:18,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:18,997 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:18,997 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688519011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688519012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688519014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688519014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,089 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/b1fa6e9c14174a27b31d2e2bcb4efbba as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/b1fa6e9c14174a27b31d2e2bcb4efbba 2023-04-05 09:54:19,093 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into b1fa6e9c14174a27b31d2e2bcb4efbba(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:19,094 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:19,094 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, priority=11, startTime=1680688458614; duration=0sec 2023-04-05 09:54:19,094 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:19,094 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:54:19,096 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:54:19,096 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C is initiating minor compaction (all files) 2023-04-05 09:54:19,096 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,096 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=55.7 K 2023-04-05 09:54:19,096 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5119d518e0d24406959000839907f897, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688453761 2023-04-05 09:54:19,097 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a2e692a5051a41ebb3c78122e0d3fc2e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680688453787 2023-04-05 09:54:19,097 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,097 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting aec64b37aa6344d69f39530964b35b69, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688454937 2023-04-05 09:54:19,098 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting fc21a1a307c548588a1f04814f28c760, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680688455605 2023-04-05 09:54:19,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,098 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 79d942576f9649f9adc7eeefa2fbbe8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457752 2023-04-05 09:54:19,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,101 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,109 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#C#compaction#135 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c94577e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,116 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,119 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,119 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,122 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,123 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,126 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:19,126 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,126 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,127 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/25a5500cb3194e13b9803c5e10085c3f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/25a5500cb3194e13b9803c5e10085c3f 2023-04-05 09:54:19,138 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 25a5500cb3194e13b9803c5e10085c3f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:19,138 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:19,138 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, priority=11, startTime=1680688458614; duration=0sec 2023-04-05 09:54:19,138 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:19,153 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/2430abc762f04834a481ccc1574242aa as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2430abc762f04834a481ccc1574242aa 2023-04-05 09:54:19,169 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/7d38de43378c4b8aab24845db98f4e1d 2023-04-05 09:54:19,172 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 2430abc762f04834a481ccc1574242aa(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:19,173 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:19,173 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, priority=11, startTime=1680688458614; duration=0sec 2023-04-05 09:54:19,173 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:19,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688519212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,226 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,230 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,238 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fdefd30[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,240 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,240 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,241 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,241 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,247 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,247 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,249 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,251 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:19,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,254 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,254 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688519315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688519315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688519318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688519319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,355 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,360 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,367 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,367 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be207b3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,371 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,371 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,375 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,378 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,378 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,378 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,378 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,378 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,381 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,381 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,381 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,389 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,390 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,390 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,490 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,494 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,497 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,497 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,498 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,498 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5e8d56[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,505 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,514 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,514 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,525 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,528 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,528 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,528 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,528 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,529 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,529 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,529 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/ef523c3be31b485d8988861a86569339 2023-04-05 09:54:19,629 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,632 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,634 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,636 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,638 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,638 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315f1664[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,646 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,646 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,648 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,648 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,653 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,653 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,653 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,753 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,759 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,764 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,764 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,764 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,764 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,765 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,768 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,768 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,768 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b04a02[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,774 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,777 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,781 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,781 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,790 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,790 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688519819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688519821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688519822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688519825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,890 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:19,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:19,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:19,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:19,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:19,894 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:19,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:19,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:19,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,896 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,897 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,897 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:19,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:19,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:19,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:19,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:19,900 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,900 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:19,900 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:19,900 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:19,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:19,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32340718[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:19,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:19,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,905 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:19,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:19,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:19,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:19,908 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:19,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:19,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:19,912 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:19,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:19,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:19,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:19,912 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:19,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:19,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:19,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:19,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:19,925 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:19,925 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:19,925 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:19,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:19,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/654b1597938a485282d09e4ec8f88fb8 2023-04-05 09:54:20,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/7d38de43378c4b8aab24845db98f4e1d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d 2023-04-05 09:54:20,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d, entries=200, sequenceid=103, filesize=31.9 K 2023-04-05 09:54:20,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/ef523c3be31b485d8988861a86569339 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339 2023-04-05 09:54:20,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339, entries=150, sequenceid=103, filesize=11.6 K 2023-04-05 09:54:20,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/654b1597938a485282d09e4ec8f88fb8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8 2023-04-05 09:54:20,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8, entries=150, sequenceid=103, filesize=11.6 K 2023-04-05 09:54:20,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1332ms, sequenceid=103, compaction requested=false 2023-04-05 09:54:20,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:20,025 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:20,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:20,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:20,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:20,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:20,029 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:20,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:20,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:20,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:20,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:20,034 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,034 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:20,035 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:20,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,054 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055c1fc6818a79485c9cf64f52a035c108 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055c1fc6818a79485c9cf64f52a035c108 2023-04-05 09:54:20,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:20,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:20,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688520265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688520369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,463 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/a7b79184a523483992d83dc884c87602 2023-04-05 09:54:20,475 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/cf399a5dc30f42e9a6d1ed9826321224 2023-04-05 09:54:20,488 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/4cfe3150735944269f4ea4f8281fbbcf 2023-04-05 09:54:20,494 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/a7b79184a523483992d83dc884c87602 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602 2023-04-05 09:54:20,498 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602, entries=150, sequenceid=119, filesize=25.1 K 2023-04-05 09:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/cf399a5dc30f42e9a6d1ed9826321224 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224 2023-04-05 09:54:20,505 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224, entries=150, sequenceid=119, filesize=11.6 K 2023-04-05 09:54:20,505 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/4cfe3150735944269f4ea4f8281fbbcf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf 2023-04-05 09:54:20,510 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf, entries=150, sequenceid=119, filesize=11.6 K 2023-04-05 09:54:20,510 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 476ms, sequenceid=119, compaction requested=true 2023-04-05 09:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:20,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,518 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,518 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596b0835[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:20,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:20,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:20,526 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,526 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:20,527 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,527 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:20,527 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,527 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:20,532 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:20,532 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:20,532 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:20,532 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:20,532 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:20,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,537 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,543 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,543 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:20,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:20,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:20,545 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:20,545 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:20,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:20,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:20,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:20,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688520602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,645 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:20,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:20,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:20,648 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:20,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:20,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:20,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:20,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:20,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:20,652 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,652 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:20,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:20,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:20,655 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ced94a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:20,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:20,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,658 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:20,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:20,660 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:20,660 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,660 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,660 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:20,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:20,663 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:20,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:20,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:20,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:20,663 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:20,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:20,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:20,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,671 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:20,671 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:20,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:20,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688520707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,771 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:20,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:20,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:20,774 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:20,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:20,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:20,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:20,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:20,778 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,778 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:20,778 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:20,778 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:20,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:20,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:20,780 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc5610b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:20,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:20,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:20,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:20,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:20,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:20,786 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:20,788 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:20,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:20,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:20,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:20,788 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:20,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,790 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,803 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:20,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:20,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:20,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,804 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:20,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,804 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:20,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688520823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688520829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688520831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688520833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,904 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:20,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:20,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:20,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:20,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:20,907 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:20,907 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,907 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:20,907 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:20,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,909 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:20,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:20,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:20,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:20,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:20,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:20,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688520911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695bb5ac[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:20,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:20,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,916 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:20,916 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,916 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:20,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:20,920 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:20,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:20,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:20,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:20,921 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:20,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:20,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,923 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:20,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:20,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:20,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:20,927 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:20,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:20,927 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:20,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:20,994 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cd035bea39c84ec39a9c5b4db82c7363 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd035bea39c84ec39a9c5b4db82c7363 2023-04-05 09:54:21,000 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/597fcab4eb534d6f85350166a778ee56 2023-04-05 09:54:21,027 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,030 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,039 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0f472f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,053 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,053 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,055 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,057 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,061 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,061 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,162 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,165 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,173 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4d7f97[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,175 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,175 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,177 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,181 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,181 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,189 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,190 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,190 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,190 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:21,190 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,190 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,190 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,190 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:21,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688521215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,290 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,294 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,297 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,300 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,300 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,300 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,300 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@344306e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,312 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,312 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,312 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,312 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,312 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,320 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,320 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/7512c593b5594382b5b9caf61d45f72f 2023-04-05 09:54:21,420 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,424 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,428 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,428 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,428 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,428 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,430 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185bee93[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,435 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,435 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,438 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,438 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,444 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,445 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,545 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,548 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,554 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,554 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,554 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,554 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc9e488[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,566 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,569 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,569 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,573 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:21,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,576 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,576 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,676 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,679 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,679 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,679 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,679 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,681 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,681 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,681 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,682 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,682 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,682 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,683 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d798541[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,695 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,699 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,700 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:21,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,706 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,706 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688521719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,806 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,810 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,813 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,814 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:21,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:21,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1c15c0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:21,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:21,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:21,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:21,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:21,823 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:21,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:21,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:21,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:21,823 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:21,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:21,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:21,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/14daa788408f430f9253d3f6e089a213 2023-04-05 09:54:21,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:21,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:21,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,831 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:21,831 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:21,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:21,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/597fcab4eb534d6f85350166a778ee56 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56 2023-04-05 09:54:21,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:21,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56, entries=200, sequenceid=139, filesize=32.1 K 2023-04-05 09:54:21,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/7512c593b5594382b5b9caf61d45f72f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f 2023-04-05 09:54:21,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f, entries=150, sequenceid=139, filesize=11.8 K 2023-04-05 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/14daa788408f430f9253d3f6e089a213 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213 2023-04-05 09:54:21,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213, entries=150, sequenceid=139, filesize=11.8 K 2023-04-05 09:54:21,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1270ms, sequenceid=139, compaction requested=true 2023-04-05 09:54:21,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:21,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:21,849 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:21,849 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:21,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:21,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:21,851 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117236 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:21,852 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A is initiating minor compaction (all files) 2023-04-05 09:54:21,852 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,852 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2430abc762f04834a481ccc1574242aa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=114.5 K 2023-04-05 09:54:21,852 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2430abc762f04834a481ccc1574242aa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457752 2023-04-05 09:54:21,853 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7d38de43378c4b8aab24845db98f4e1d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680688458078 2023-04-05 09:54:21,853 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a7b79184a523483992d83dc884c87602, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680688458699 2023-04-05 09:54:21,853 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:21,853 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B is initiating minor compaction (all files) 2023-04-05 09:54:21,853 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,854 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/b1fa6e9c14174a27b31d2e2bcb4efbba, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=46.7 K 2023-04-05 09:54:21,854 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 597fcab4eb534d6f85350166a778ee56, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:21,854 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b1fa6e9c14174a27b31d2e2bcb4efbba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457752 2023-04-05 09:54:21,854 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ef523c3be31b485d8988861a86569339, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680688458078 2023-04-05 09:54:21,855 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cf399a5dc30f42e9a6d1ed9826321224, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680688458699 2023-04-05 09:54:21,855 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 7512c593b5594382b5b9caf61d45f72f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:21,873 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#A#compaction#144 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:21,874 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#B#compaction#145 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:21,932 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:21,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:21,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:21,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:21,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:21,935 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:21,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:21,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:21,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:21,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:21,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:21,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:21,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:21,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:21,939 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:21,939 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:21,939 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:21,939 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:21,954 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304057f265d4d73e34fd58f8cd3fca2bd3591 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057f265d4d73e34fd58f8cd3fca2bd3591 2023-04-05 09:54:22,299 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/2dc2a6982bca4d30a9aeed546603d3e9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2dc2a6982bca4d30a9aeed546603d3e9 2023-04-05 09:54:22,309 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 2dc2a6982bca4d30a9aeed546603d3e9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:22,309 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:22,310 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, priority=12, startTime=1680688461849; duration=0sec 2023-04-05 09:54:22,310 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:22,310 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:22,312 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:22,312 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C is initiating minor compaction (all files) 2023-04-05 09:54:22,312 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:22,312 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/25a5500cb3194e13b9803c5e10085c3f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=46.7 K 2023-04-05 09:54:22,312 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 25a5500cb3194e13b9803c5e10085c3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688457752 2023-04-05 09:54:22,313 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 654b1597938a485282d09e4ec8f88fb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680688458078 2023-04-05 09:54:22,313 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4cfe3150735944269f4ea4f8281fbbcf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680688458699 2023-04-05 09:54:22,314 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 14daa788408f430f9253d3f6e089a213, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:22,317 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/c257e24a5e1745e6b970214af8eb2d12 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c257e24a5e1745e6b970214af8eb2d12 2023-04-05 09:54:22,322 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into c257e24a5e1745e6b970214af8eb2d12(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:22,323 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:22,323 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, priority=12, startTime=1680688461849; duration=0sec 2023-04-05 09:54:22,323 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:22,323 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#C#compaction#147 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:22,337 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/212720d6e30640dca6cce01e5dbe303d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/212720d6e30640dca6cce01e5dbe303d 2023-04-05 09:54:22,342 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 212720d6e30640dca6cce01e5dbe303d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:22,342 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:22,342 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, priority=12, startTime=1680688461849; duration=0sec 2023-04-05 09:54:22,342 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:22,359 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/d39b25b1131b4890803e75288e6bb712 2023-04-05 09:54:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:22,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:22,770 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/370ca64f3e4e46598bac10384f768444 2023-04-05 09:54:22,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688522780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,787 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/0f78df38b7a946e190422b98bcd59fc4 2023-04-05 09:54:22,793 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/d39b25b1131b4890803e75288e6bb712 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712 2023-04-05 09:54:22,798 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712, entries=150, sequenceid=151, filesize=25.3 K 2023-04-05 09:54:22,803 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/370ca64f3e4e46598bac10384f768444 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444 2023-04-05 09:54:22,808 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444, entries=150, sequenceid=151, filesize=11.8 K 2023-04-05 09:54:22,809 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/0f78df38b7a946e190422b98bcd59fc4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4 2023-04-05 09:54:22,814 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4, entries=150, sequenceid=151, filesize=11.8 K 2023-04-05 09:54:22,815 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 876ms, sequenceid=151, compaction requested=false 2023-04-05 09:54:22,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:22,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:22,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f44fc6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:22,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:22,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,821 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:22,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:22,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:22,826 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:22,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:22,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:22,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:22,826 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:22,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:22,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:22,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:22,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,834 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:22,834 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:22,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:22,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:22,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:22,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:22,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688522857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,862 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056c37920026bc4496a97852337c1c73e8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056c37920026bc4496a97852337c1c73e8 2023-04-05 09:54:22,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688522860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688522860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688522862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688522884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,935 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:22,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:22,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:22,938 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:22,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:22,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:22,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:22,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:22,940 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:22,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:22,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:22,942 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:22,942 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:22,942 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:22,942 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:22,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bea1df4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:22,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:22,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:22,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:22,949 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:22,951 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:22,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:22,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:22,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:22,952 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:22,957 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,957 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:22,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,957 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:22,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:22,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,959 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:22,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:22,961 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:22,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:22,961 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:22,961 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:22,961 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:22,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:22,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:22,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688522963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688522965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688522965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688522965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,062 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,065 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,067 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,069 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,069 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,069 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,069 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,071 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac8c645[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,077 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,081 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,081 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,083 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,083 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,083 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,084 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,086 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688523087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,089 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,089 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688523165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688523168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688523169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688523169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,189 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,192 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,193 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,193 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,195 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,197 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,197 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,197 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,197 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,198 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,198 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25856ac9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,200 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,200 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,207 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,207 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,207 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,207 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,207 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,210 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,210 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,210 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,211 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,213 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,215 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,215 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,216 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,216 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,270 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=175, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/af34b05bcc954c219758d46ac8148541 2023-04-05 09:54:23,316 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,319 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,321 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,321 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,321 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,321 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,324 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f758bc0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,326 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,326 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,329 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,337 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,337 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,340 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,351 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,351 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,351 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,351 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688523390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,451 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,454 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,456 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,456 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,456 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,458 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,458 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,458 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,458 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,459 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240b2867[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,467 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,468 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,469 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,469 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688523469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,469 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,471 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,472 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688523471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,475 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,475 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,475 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,476 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688523476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688523476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,576 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,579 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,582 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,583 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,585 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ed284d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,587 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,589 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,592 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,592 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,602 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,602 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,603 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,603 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,603 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/f3349f4a4f5e4e77b3d1cc8418d2dd56 2023-04-05 09:54:23,703 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,707 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,709 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,711 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,711 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,711 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,711 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,712 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b0cebb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,718 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,720 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,721 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,723 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,724 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,732 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,732 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,732 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,732 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,833 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,839 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,839 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,839 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,839 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,856 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,856 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,859 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e86826[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:23,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:23,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:23,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:23,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:23,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:23,869 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:23,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:23,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:23,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:23,869 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:23,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:23,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:23,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:23,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:23,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:23,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:23,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:23,879 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:23,879 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:23,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:23,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:23,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51608 deadline: 1680688523894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688523970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688523978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,979 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:23,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:23,984 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:23,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:23,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:23,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688523982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688523982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:23,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:23,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:23,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:23,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:23,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:23,995 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:23,996 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,996 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:23,996 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:23,996 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:23,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:23,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,999 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:23,999 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385d38ec[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:24,007 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:24,007 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:24,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:24,012 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:24,016 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:24,016 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:24,016 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:24,016 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:24,016 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:24,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:24,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:24,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,023 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:24,023 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:24,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/c66e135ed14f484190a827803d7af1b0 2023-04-05 09:54:24,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/af34b05bcc954c219758d46ac8148541 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541 2023-04-05 09:54:24,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541, entries=200, sequenceid=175, filesize=32.1 K 2023-04-05 09:54:24,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/f3349f4a4f5e4e77b3d1cc8418d2dd56 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56 2023-04-05 09:54:24,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56, entries=150, sequenceid=175, filesize=11.8 K 2023-04-05 09:54:24,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/c66e135ed14f484190a827803d7af1b0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0 2023-04-05 09:54:24,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0, entries=150, sequenceid=175, filesize=11.8 K 2023-04-05 09:54:24,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1267ms, sequenceid=175, compaction requested=true 2023-04-05 09:54:24,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,112 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:24,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:24,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:24,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:24,112 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:24,113 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:24,114 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A is initiating minor compaction (all files) 2023-04-05 09:54:24,114 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,114 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c257e24a5e1745e6b970214af8eb2d12, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=83.0 K 2023-04-05 09:54:24,115 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:24,115 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c257e24a5e1745e6b970214af8eb2d12, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:24,115 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B is initiating minor compaction (all files) 2023-04-05 09:54:24,115 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,115 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2dc2a6982bca4d30a9aeed546603d3e9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=35.6 K 2023-04-05 09:54:24,115 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d39b25b1131b4890803e75288e6bb712, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680688460588 2023-04-05 09:54:24,116 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2dc2a6982bca4d30a9aeed546603d3e9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:24,116 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting af34b05bcc954c219758d46ac8148541, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462753 2023-04-05 09:54:24,116 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 370ca64f3e4e46598bac10384f768444, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680688460588 2023-04-05 09:54:24,117 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f3349f4a4f5e4e77b3d1cc8418d2dd56, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462771 2023-04-05 09:54:24,123 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:24,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:24,129 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#B#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:24,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:24,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:24,131 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:24,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:24,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:24,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,134 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:24,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:24,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:24,148 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:24,148 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:24,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:24,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:24,151 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#A#compaction#154 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:24,165 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/c41fc1b626f141c69dd62f4afc74eadc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/c41fc1b626f141c69dd62f4afc74eadc 2023-04-05 09:54:24,171 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into c41fc1b626f141c69dd62f4afc74eadc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:24,172 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,172 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, priority=13, startTime=1680688464112; duration=0sec 2023-04-05 09:54:24,172 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:24,172 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:24,173 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:24,173 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C is initiating minor compaction (all files) 2023-04-05 09:54:24,173 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,174 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/212720d6e30640dca6cce01e5dbe303d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=35.6 K 2023-04-05 09:54:24,174 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 212720d6e30640dca6cce01e5dbe303d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680688460257 2023-04-05 09:54:24,175 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0f78df38b7a946e190422b98bcd59fc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680688460588 2023-04-05 09:54:24,175 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c66e135ed14f484190a827803d7af1b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462771 2023-04-05 09:54:24,189 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#C#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:24,195 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d6105cf0d4a34b45a9cd87e7288c80cb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6105cf0d4a34b45a9cd87e7288c80cb 2023-04-05 09:54:24,207 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/52f382577aaf416fa2ebab3a50010759 2023-04-05 09:54:24,210 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/722955dd6a994b6da587ebfd53b498aa as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/722955dd6a994b6da587ebfd53b498aa 2023-04-05 09:54:24,214 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 722955dd6a994b6da587ebfd53b498aa(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:24,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,215 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, priority=13, startTime=1680688464112; duration=0sec 2023-04-05 09:54:24,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:24,600 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/beed3c0399a64803be37f68a2c0b34a6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/beed3c0399a64803be37f68a2c0b34a6 2023-04-05 09:54:24,607 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into beed3c0399a64803be37f68a2c0b34a6(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:24,607 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,607 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, priority=13, startTime=1680688464112; duration=0sec 2023-04-05 09:54:24,607 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:24,621 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/963d048ad2084b1bbeb0b21b0551036d 2023-04-05 09:54:24,661 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/19fc394cfa1d4baaae58a9e2a5f8c3cf 2023-04-05 09:54:24,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/52f382577aaf416fa2ebab3a50010759 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759 2023-04-05 09:54:24,673 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759, entries=150, sequenceid=186, filesize=25.3 K 2023-04-05 09:54:24,674 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/963d048ad2084b1bbeb0b21b0551036d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d 2023-04-05 09:54:24,678 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d, entries=150, sequenceid=186, filesize=11.8 K 2023-04-05 09:54:24,679 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/19fc394cfa1d4baaae58a9e2a5f8c3cf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf 2023-04-05 09:54:24,685 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf, entries=150, sequenceid=186, filesize=11.8 K 2023-04-05 09:54:24,686 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 538ms, sequenceid=186, compaction requested=false 2023-04-05 09:54:24,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,686 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:24,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6923c0ea[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:24,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:24,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:24,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:24,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:24,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:24,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:24,703 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:24,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:24,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:24,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:24,704 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:24,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:24,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:24,715 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,715 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,815 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:24,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:24,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:24,819 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:24,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:24,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:24,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,821 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:24,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:24,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:24,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:24,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:24,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,834 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,834 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,835 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59561e63[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:24,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:24,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:24,838 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:24,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:24,838 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:24,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:24,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,840 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:24,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:24,842 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:24,859 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:24,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:24,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:24,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:24,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,871 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:24,871 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:24,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:24,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:24,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,971 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:24,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:24,980 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:24,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:24,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:24,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,982 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:24,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:24,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:24,989 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:24,989 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:24,991 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:24,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:25,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:25,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:25,029 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405085c39238f344b2f925a628995428b6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405085c39238f344b2f925a628995428b6c 2023-04-05 09:54:25,048 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/6ed38db042be492aa7f8b2d29a4985a8 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680688465035, max timestamp=1680688465079Num uniques 859; 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680688465035, max timestamp=1680688465079Num uniques 859; 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680688465035, max timestamp=1680688465079Num uniques 859; 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-05 09:54:25,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-05 09:54:25,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688525097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688525098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680688465093, max timestamp=1680688465101Num uniques 1318; 2023-04-05 09:54:25,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=60358, new segment data size=14558, suffix heap size=165040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1318, new segment cells count=318 2023-04-05 09:54:25,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688465093, max timestamp=1680688465101Num uniques 1318; 2023-04-05 09:54:25,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=60358, new segment data size=6870, suffix heap size=165040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1318, new segment cells count=150 2023-04-05 09:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688465093, max timestamp=1680688465101Num uniques 1318; 2023-04-05 09:54:25,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=60358, new segment data size=6870, suffix heap size=165040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1318, new segment cells count=150 2023-04-05 09:54:25,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680688465119, max timestamp=1680688465124Num uniques 1009; 2023-04-05 09:54:25,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=14972, suffix heap size=132904,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=327 2023-04-05 09:54:25,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680688465119, max timestamp=1680688465124Num uniques 1009; 2023-04-05 09:54:25,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=14972, suffix heap size=132904,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=327 2023-04-05 09:54:25,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680688465119, max timestamp=1680688465124Num uniques 1177; 2023-04-05 09:54:25,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680688465126, max timestamp=1680688465143Num uniques 1186; 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=19966, suffix heap size=151312,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=436 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680688465126, max timestamp=1680688465143Num uniques 1186; 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680688465126, max timestamp=1680688465143Num uniques 1186; 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=19966, suffix heap size=151312,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=436 2023-04-05 09:54:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=19966, suffix heap size=151312,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=436 2023-04-05 09:54:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680688465148, max timestamp=1680688465160Num uniques 1295; 2023-04-05 09:54:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=20380, suffix heap size=162648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=445 2023-04-05 09:54:25,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680688465148, max timestamp=1680688465160Num uniques 1295; 2023-04-05 09:54:25,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=20380, suffix heap size=162648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=445 2023-04-05 09:54:25,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680688465148, max timestamp=1680688465160Num uniques 1295; 2023-04-05 09:54:25,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=20380, suffix heap size=162648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=445 2023-04-05 09:54:25,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688525178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688525179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688525180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688525203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688525203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688525281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688525283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688525283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688525406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688525407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,485 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/07765b33f3c4441db3e92309c7aed4f9 2023-04-05 09:54:25,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688525484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688525486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688525486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,499 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/599cbd43876f4b72b5fc45a4fa911ed2 2023-04-05 09:54:25,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/6ed38db042be492aa7f8b2d29a4985a8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8 2023-04-05 09:54:25,508 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8, entries=100, sequenceid=195, filesize=18.5 K 2023-04-05 09:54:25,509 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/07765b33f3c4441db3e92309c7aed4f9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9 2023-04-05 09:54:25,513 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9, entries=50, sequenceid=195, filesize=7.1 K 2023-04-05 09:54:25,514 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/599cbd43876f4b72b5fc45a4fa911ed2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2 2023-04-05 09:54:25,518 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2, entries=50, sequenceid=195, filesize=7.1 K 2023-04-05 09:54:25,519 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 529ms, sequenceid=195, compaction requested=true 2023-04-05 09:54:25,519 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:25,519 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:25,519 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:25,519 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:25,519 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,519 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:25,519 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:25,521 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:25,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:25,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff9f437[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:25,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:25,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,524 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:25,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:25,526 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,526 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:25,526 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:25,526 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:25,533 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:25,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:25,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:25,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:25,533 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:25,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:25,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:25,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,543 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:25,543 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:25,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:25,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:25,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:25,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,544 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:25,544 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:25,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:25,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:25,644 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:25,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:25,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:25,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:25,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:25,649 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:25,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:25,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:25,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:25,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:25,651 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:25,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:25,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:25,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:25,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:25,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:25,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:25,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:25,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:25,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:25,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:25,653 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=174.43 KB heapSize=477.70 KB 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:25,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:25,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:25,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688525715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688525715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51628 deadline: 1680688525789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51636 deadline: 1680688525789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688525790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688525818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:25,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688525818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51612 deadline: 1680688526021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51616 deadline: 1680688526021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,070 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c7a0b1f73956475183444a995e25884d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c7a0b1f73956475183444a995e25884d 2023-04-05 09:54:26,077 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/af272f0237874069ba2947635e846307 2023-04-05 09:54:26,092 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/5e086f458dee46659375fa07e5403511 2023-04-05 09:54:26,102 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/6d8031a42bac4148acf8fb128e2bec5b 2023-04-05 09:54:26,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/af272f0237874069ba2947635e846307 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307 2023-04-05 09:54:26,112 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307, entries=150, sequenceid=301, filesize=25.4 K 2023-04-05 09:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/5e086f458dee46659375fa07e5403511 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511 2023-04-05 09:54:26,117 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511, entries=150, sequenceid=301, filesize=11.9 K 2023-04-05 09:54:26,122 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/6d8031a42bac4148acf8fb128e2bec5b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b 2023-04-05 09:54:26,136 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b, entries=150, sequenceid=301, filesize=11.9 K 2023-04-05 09:54:26,137 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~477.66 KB/489120, currentSize=13.42 KB/13740 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 484ms, sequenceid=301, compaction requested=true 2023-04-05 09:54:26,137 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:26,137 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:26,140 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e322d7c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:26,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:26,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:26,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:26,145 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:26,148 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:26,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:26,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:26,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:26,149 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:26,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:26,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:26,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,158 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:26,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:26,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:26,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:26,162 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:26,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:26,162 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:26,163 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:26,263 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:26,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:26,267 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:26,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:26,270 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:26,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:26,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:26,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:26,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:26,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:26,273 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:26,273 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:26,273 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-05 09:54:26,273 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:26,273 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:26,273 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:26,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:26,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:26,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:26,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:26,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688466323, max timestamp=1680688466328Num uniques 859; 2023-04-05 09:54:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688466323, max timestamp=1680688466328Num uniques 859; 2023-04-05 09:54:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688466323, max timestamp=1680688466328Num uniques 859; 2023-04-05 09:54:26,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:26,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688466335, max timestamp=1680688466343Num uniques 1068; 2023-04-05 09:54:26,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:54:26,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688466335, max timestamp=1680688466343Num uniques 1068; 2023-04-05 09:54:26,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:54:26,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680688466335, max timestamp=1680688466343Num uniques 1068; 2023-04-05 09:54:26,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-05 09:54:26,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688466359, max timestamp=1680688466364Num uniques 1227; 2023-04-05 09:54:26,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-05 09:54:26,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688466359, max timestamp=1680688466364Num uniques 1227; 2023-04-05 09:54:26,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-05 09:54:26,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688466359, max timestamp=1680688466364Num uniques 1227; 2023-04-05 09:54:26,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-05 09:54:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680688466377, max timestamp=1680688466382Num uniques 1036; 2023-04-05 09:54:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680688466377, max timestamp=1680688466382Num uniques 1036; 2023-04-05 09:54:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-05 09:54:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-05 09:54:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680688466377, max timestamp=1680688466382Num uniques 1036; 2023-04-05 09:54:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-05 09:54:26,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688466414, max timestamp=1680688466419Num uniques 1195; 2023-04-05 09:54:26,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-05 09:54:26,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688466414, max timestamp=1680688466419Num uniques 1195; 2023-04-05 09:54:26,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-05 09:54:26,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688466414, max timestamp=1680688466419Num uniques 1195; 2023-04-05 09:54:26,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-05 09:54:26,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680688466433, max timestamp=1680688466438Num uniques 1104; 2023-04-05 09:54:26,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-05 09:54:26,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680688466433, max timestamp=1680688466438Num uniques 1104; 2023-04-05 09:54:26,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-05 09:54:26,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680688466433, max timestamp=1680688466438Num uniques 1104; 2023-04-05 09:54:26,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1680688466446, max timestamp=1680688466452Num uniques 1013; 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1680688466446, max timestamp=1680688466452Num uniques 1013; 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1680688466446, max timestamp=1680688466452Num uniques 1013; 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-04-05 09:54:26,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-04-05 09:54:26,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680688466461, max timestamp=1680688466471Num uniques 1222; 2023-04-05 09:54:26,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=19322, suffix heap size=155056,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=422 2023-04-05 09:54:26,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680688466461, max timestamp=1680688466471Num uniques 1222; 2023-04-05 09:54:26,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=19322, suffix heap size=155056,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=422 2023-04-05 09:54:26,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680688466461, max timestamp=1680688466471Num uniques 1222; 2023-04-05 09:54:26,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=19322, suffix heap size=155056,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=422 2023-04-05 09:54:26,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688526488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1680688466485, max timestamp=1680688466490Num uniques 1281; 2023-04-05 09:54:26,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680688466485, max timestamp=1680688466490Num uniques 1281; 2023-04-05 09:54:26,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1680688466485, max timestamp=1680688466490Num uniques 1281; 2023-04-05 09:54:26,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=58656, new segment data size=7320, suffix heap size=161192,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=160 2023-04-05 09:54:26,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=58656, new segment data size=8276, suffix heap size=161192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=181 2023-04-05 09:54:26,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=58666, new segment data size=8286, suffix heap size=161192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=181 2023-04-05 09:54:26,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680688466499, max timestamp=1680688466510Num uniques 1040; 2023-04-05 09:54:26,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=20150, suffix heap size=136128,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=440 2023-04-05 09:54:26,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680688466499, max timestamp=1680688466510Num uniques 1040; 2023-04-05 09:54:26,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=20150, suffix heap size=136128,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=440 2023-04-05 09:54:26,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680688466499, max timestamp=1680688466510Num uniques 1019; 2023-04-05 09:54:26,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46674, new segment data size=20150, suffix heap size=133944,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=440 2023-04-05 09:54:26,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688526525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688526525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680688466522, max timestamp=1680688466527Num uniques 1299; 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680688466522, max timestamp=1680688466527Num uniques 1299; 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=11404, suffix heap size=163064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=249 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680688466522, max timestamp=1680688466527Num uniques 1299; 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=11404, suffix heap size=163064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=249 2023-04-05 09:54:26,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=11404, suffix heap size=163064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=249 2023-04-05 09:54:26,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680688466547, max timestamp=1680688466555Num uniques 1108; 2023-04-05 09:54:26,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-05 09:54:26,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680688466547, max timestamp=1680688466555Num uniques 1108; 2023-04-05 09:54:26,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-05 09:54:26,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680688466547, max timestamp=1680688466555Num uniques 1108; 2023-04-05 09:54:26,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680688466569, max timestamp=1680688466583Num uniques 1067; 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680688466569, max timestamp=1680688466583Num uniques 1067; 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680688466569, max timestamp=1680688466583Num uniques 1067; 2023-04-05 09:54:26,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 09:54:26,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680688466598, max timestamp=1680688466604Num uniques 1126; 2023-04-05 09:54:26,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-05 09:54:26,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680688466598, max timestamp=1680688466604Num uniques 1126; 2023-04-05 09:54:26,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-05 09:54:26,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680688466598, max timestamp=1680688466604Num uniques 1126; 2023-04-05 09:54:26,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680688466617, max timestamp=1680688466626Num uniques 1185; 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=13050, suffix heap size=151208,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=285 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680688466617, max timestamp=1680688466626Num uniques 1185; 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680688466617, max timestamp=1680688466626Num uniques 1185; 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=13050, suffix heap size=151208,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=285 2023-04-05 09:54:26,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=13050, suffix heap size=151208,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=285 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680688466638, max timestamp=1680688466666Num uniques 1144; 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680688466638, max timestamp=1680688466666Num uniques 1144; 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680688466638, max timestamp=1680688466666Num uniques 1144; 2023-04-05 09:54:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-05 09:54:26,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680688466674, max timestamp=1680688466689Num uniques 1203; 2023-04-05 09:54:26,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=13875, suffix heap size=153080,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=303 2023-04-05 09:54:26,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680688466674, max timestamp=1680688466689Num uniques 1203; 2023-04-05 09:54:26,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680688466674, max timestamp=1680688466689Num uniques 1203; 2023-04-05 09:54:26,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=13875, suffix heap size=153080,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=303 2023-04-05 09:54:26,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=13875, suffix heap size=153080,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=303 2023-04-05 09:54:26,700 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040590c51e0863624de684836fd27789299c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040590c51e0863624de684836fd27789299c 2023-04-05 09:54:26,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688526713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688526713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680688466712, max timestamp=1680688466715Num uniques 1162; 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680688466712, max timestamp=1680688466715Num uniques 1162; 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680688466712, max timestamp=1680688466715Num uniques 1162; 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-04-05 09:54:26,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-04-05 09:54:26,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680688466727, max timestamp=1680688466741Num uniques 1021; 2023-04-05 09:54:26,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=14696, suffix heap size=134152,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=321 2023-04-05 09:54:26,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680688466727, max timestamp=1680688466741Num uniques 1021; 2023-04-05 09:54:26,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680688466727, max timestamp=1680688466741Num uniques 1021; 2023-04-05 09:54:26,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=14696, suffix heap size=134152,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=321 2023-04-05 09:54:26,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=14696, suffix heap size=134152,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=321 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680688466760, max timestamp=1680688466765Num uniques 1180; 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680688466760, max timestamp=1680688466765Num uniques 1180; 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680688466760, max timestamp=1680688466765Num uniques 1180; 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=10530, suffix heap size=150688,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=230 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=10530, suffix heap size=150688,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=230 2023-04-05 09:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=10530, suffix heap size=150688,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=230 2023-04-05 09:54:26,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680688466781, max timestamp=1680688466790Num uniques 1089; 2023-04-05 09:54:26,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-05 09:54:26,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680688466781, max timestamp=1680688466790Num uniques 1089; 2023-04-05 09:54:26,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680688466781, max timestamp=1680688466790Num uniques 1089; 2023-04-05 09:54:26,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-05 09:54:26,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-05 09:54:26,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=448, cellSize=20518, totalHeapSize=46832, min timestamp=1680688466826, max timestamp=1680688466838Num uniques 1198; 2023-04-05 09:54:26,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=448, cellSize=20518, totalHeapSize=46832, min timestamp=1680688466826, max timestamp=1680688466838Num uniques 1198; 2023-04-05 09:54:26,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=20518, suffix heap size=152560,new segment heap size=46832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=448 2023-04-05 09:54:26,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=448, cellSize=20518, totalHeapSize=46832, min timestamp=1680688466826, max timestamp=1680688466838Num uniques 1198; 2023-04-05 09:54:26,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=20518, suffix heap size=152560,new segment heap size=46832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=448 2023-04-05 09:54:26,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=20518, suffix heap size=152560,new segment heap size=46832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=448 2023-04-05 09:54:26,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688526854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:26,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688526855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:26,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680688466848, max timestamp=1680688466859Num uniques 1307; 2023-04-05 09:54:26,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680688466848, max timestamp=1680688466859Num uniques 1307; 2023-04-05 09:54:26,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=59855, new segment data size=14055, suffix heap size=163896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1307, new segment cells count=307 2023-04-05 09:54:26,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=59855, new segment data size=14055, suffix heap size=163896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1307, new segment cells count=307 2023-04-05 09:54:26,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688466848, max timestamp=1680688466859Num uniques 1307; 2023-04-05 09:54:26,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=59855, new segment data size=6870, suffix heap size=163896,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1307, new segment cells count=150 2023-04-05 09:54:26,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680688466879, max timestamp=1680688466886Num uniques 1166; 2023-04-05 09:54:26,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680688466879, max timestamp=1680688466886Num uniques 1166; 2023-04-05 09:54:26,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=9886, suffix heap size=149232,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=216 2023-04-05 09:54:26,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=9886, suffix heap size=149232,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=216 2023-04-05 09:54:26,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680688466879, max timestamp=1680688466886Num uniques 1009; 2023-04-05 09:54:26,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=9886, suffix heap size=132904,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=216 2023-04-05 09:54:26,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680688466910, max timestamp=1680688466918Num uniques 1075; 2023-04-05 09:54:26,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-05 09:54:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680688466910, max timestamp=1680688466918Num uniques 1075; 2023-04-05 09:54:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680688466910, max timestamp=1680688466918Num uniques 1075; 2023-04-05 09:54:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-05 09:54:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-05 09:54:26,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680688466935, max timestamp=1680688466942Num uniques 1084; 2023-04-05 09:54:26,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-05 09:54:26,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680688466935, max timestamp=1680688466942Num uniques 1084; 2023-04-05 09:54:26,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680688466935, max timestamp=1680688466942Num uniques 1084; 2023-04-05 09:54:26,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-05 09:54:26,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-05 09:54:27,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688526968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8818, totalHeapSize=20312, min timestamp=1680688466965, max timestamp=1680688467001Num uniques 1093; 2023-04-05 09:54:27,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50038, new segment data size=8818, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-05 09:54:27,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8833, totalHeapSize=20312, min timestamp=1680688466965, max timestamp=1680688467001Num uniques 1093; 2023-04-05 09:54:27,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50053, new segment data size=8833, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-05 09:54:27,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8834, totalHeapSize=20312, min timestamp=1680688466965, max timestamp=1680688467001Num uniques 1093; 2023-04-05 09:54:27,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50054, new segment data size=8834, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680688467029, max timestamp=1680688467040Num uniques 1052; 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=11540, suffix heap size=137376,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=252 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680688467029, max timestamp=1680688467040Num uniques 1052; 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680688467029, max timestamp=1680688467040Num uniques 1052; 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=11540, suffix heap size=137376,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=252 2023-04-05 09:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=11540, suffix heap size=137376,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=252 2023-04-05 09:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680688467054, max timestamp=1680688467063Num uniques 1111; 2023-04-05 09:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680688467054, max timestamp=1680688467063Num uniques 1111; 2023-04-05 09:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=11946, suffix heap size=143512,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=261 2023-04-05 09:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=11946, suffix heap size=143512,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=261 2023-04-05 09:54:27,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688467060, max timestamp=1680688467063Num uniques 1111; 2023-04-05 09:54:27,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=6870, suffix heap size=143512,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=150 2023-04-05 09:54:27,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7775, totalHeapSize=17920, min timestamp=1680688467072, max timestamp=1680688467084Num uniques 1120; 2023-04-05 09:54:27,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51285, new segment data size=7775, suffix heap size=144448,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=170 2023-04-05 09:54:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7775, totalHeapSize=17920, min timestamp=1680688467072, max timestamp=1680688467084Num uniques 1120; 2023-04-05 09:54:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7772, totalHeapSize=17920, min timestamp=1680688467072, max timestamp=1680688467084Num uniques 1009; 2023-04-05 09:54:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51285, new segment data size=7775, suffix heap size=144448,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=170 2023-04-05 09:54:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7772, suffix heap size=132904,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=170 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680688467098, max timestamp=1680688467107Num uniques 1029; 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680688467098, max timestamp=1680688467107Num uniques 1029; 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680688467098, max timestamp=1680688467107Num uniques 1029; 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-05 09:54:27,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-05 09:54:27,114 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/12dd493b10d04b5bbe7375e62e0ed32c 2023-04-05 09:54:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1680688467119, max timestamp=1680688467136Num uniques 1088; 2023-04-05 09:54:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13186, totalHeapSize=30192, min timestamp=1680688467119, max timestamp=1680688467136Num uniques 1088; 2023-04-05 09:54:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=13188, suffix heap size=141120,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=288 2023-04-05 09:54:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49826, new segment data size=13186, suffix heap size=141120,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=288 2023-04-05 09:54:27,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13178, totalHeapSize=30192, min timestamp=1680688467119, max timestamp=1680688467136Num uniques 1088; 2023-04-05 09:54:27,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49818, new segment data size=13178, suffix heap size=141120,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=288 2023-04-05 09:54:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680688467146, max timestamp=1680688467153Num uniques 1147; 2023-04-05 09:54:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-04-05 09:54:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680688467146, max timestamp=1680688467153Num uniques 1147; 2023-04-05 09:54:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-04-05 09:54:27,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680688467146, max timestamp=1680688467153Num uniques 1147; 2023-04-05 09:54:27,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-04-05 09:54:27,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1680688467158, max timestamp=1680688467169Num uniques 1156; 2023-04-05 09:54:27,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-04-05 09:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1680688467158, max timestamp=1680688467169Num uniques 1156; 2023-04-05 09:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1680688467158, max timestamp=1680688467169Num uniques 1156; 2023-04-05 09:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-04-05 09:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-04-05 09:54:27,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688527184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=315, cellSize=14420, totalHeapSize=33000, min timestamp=1680688467174, max timestamp=1680688467185Num uniques 1215; 2023-04-05 09:54:27,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=14420, suffix heap size=154328,new segment heap size=33000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=315 2023-04-05 09:54:27,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=315, cellSize=14420, totalHeapSize=33000, min timestamp=1680688467174, max timestamp=1680688467185Num uniques 1215; 2023-04-05 09:54:27,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=14420, suffix heap size=154328,new segment heap size=33000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=315 2023-04-05 09:54:27,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=315, cellSize=14420, totalHeapSize=33000, min timestamp=1680688467174, max timestamp=1680688467185Num uniques 1215; 2023-04-05 09:54:27,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=14420, suffix heap size=154328,new segment heap size=33000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=315 2023-04-05 09:54:27,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680688467222, max timestamp=1680688467232Num uniques 1174; 2023-04-05 09:54:27,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53764, new segment data size=10254, suffix heap size=150064,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1174, new segment cells count=224 2023-04-05 09:54:27,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680688467222, max timestamp=1680688467232Num uniques 1174; 2023-04-05 09:54:27,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680688467222, max timestamp=1680688467232Num uniques 1174; 2023-04-05 09:54:27,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53764, new segment data size=10254, suffix heap size=150064,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1174, new segment cells count=224 2023-04-05 09:54:27,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53764, new segment data size=10254, suffix heap size=150064,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1174, new segment cells count=224 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1680688467241, max timestamp=1680688467250Num uniques 1083; 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8372, totalHeapSize=19272, min timestamp=1680688467241, max timestamp=1680688467250Num uniques 1083; 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8368, totalHeapSize=19272, min timestamp=1680688467241, max timestamp=1680688467250Num uniques 1083; 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49588, new segment data size=8368, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-05 09:54:27,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49592, new segment data size=8372, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-05 09:54:27,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=592, cellSize=27112, totalHeapSize=61808, min timestamp=1680688467258, max timestamp=1680688467271Num uniques 1042; 2023-04-05 09:54:27,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=27112, suffix heap size=136336,new segment heap size=61808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=592 2023-04-05 09:54:27,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=592, cellSize=27112, totalHeapSize=61808, min timestamp=1680688467258, max timestamp=1680688467271Num uniques 1042; 2023-04-05 09:54:27,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=27112, suffix heap size=136336,new segment heap size=61808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=592 2023-04-05 09:54:27,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=592, cellSize=27112, totalHeapSize=61808, min timestamp=1680688467258, max timestamp=1680688467271Num uniques 1042; 2023-04-05 09:54:27,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=27112, suffix heap size=136336,new segment heap size=61808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=592 2023-04-05 09:54:27,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688527287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688527287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688527318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688527318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688527318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688527419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688527419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688527425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688527425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688527427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,528 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/e7ca8f19ee4a47cfbb6e3b1f73068e49 2023-04-05 09:54:27,539 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/55e083563ea44b5ab34e15ac10871687 2023-04-05 09:54:27,546 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/12dd493b10d04b5bbe7375e62e0ed32c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c 2023-04-05 09:54:27,550 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c, entries=50, sequenceid=306, filesize=11.7 K 2023-04-05 09:54:27,551 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/e7ca8f19ee4a47cfbb6e3b1f73068e49 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49 2023-04-05 09:54:27,555 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49, entries=50, sequenceid=306, filesize=7.1 K 2023-04-05 09:54:27,556 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/55e083563ea44b5ab34e15ac10871687 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687 2023-04-05 09:54:27,560 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687, entries=50, sequenceid=306, filesize=7.1 K 2023-04-05 09:54:27,561 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=194.56 KB/199230 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1288ms, sequenceid=306, compaction requested=true 2023-04-05 09:54:27,561 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:27,561 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:27,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@796188c0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:27,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:27,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:27,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:27,572 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:27,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:27,574 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:27,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:27,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:27,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:27,574 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:27,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:27,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,582 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:27,582 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=194.56 KB heapSize=523.69 KB 2023-04-05 09:54:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688527622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688527623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:27,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:27,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688527628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688527629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688527633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,641 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f2478dc21b974b7cadb52956e9e871f7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2478dc21b974b7cadb52956e9e871f7 2023-04-05 09:54:27,682 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:27,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:27,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:27,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:27,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:27,686 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,688 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:27,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:27,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:27,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:27,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:27,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ad3489[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:27,694 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:27,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:27,695 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:27,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:27,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:27,700 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:27,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:27,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:27,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:27,700 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:27,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:27,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,707 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:27,707 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:27,807 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:27,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:27,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:27,811 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,812 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:27,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:27,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:27,814 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,814 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:27,814 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:27,814 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:27,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e532e7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:27,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:27,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,822 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:27,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:27,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:27,826 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:27,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:27,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:27,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:27,826 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:27,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:27,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,833 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:27,834 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:27,834 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:27,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688527926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688527928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688527929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,934 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:27,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688527933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:27,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:27,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:27,937 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:27,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:27,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:27,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,939 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:27,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688527938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:27,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:27,941 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,941 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:27,941 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:27,941 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:27,942 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa3b64a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:27,944 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:27,944 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:27,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:27,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:27,953 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:27,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:27,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:27,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:27,953 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:27,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:27,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:27,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:27,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:27,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:27,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:27,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:27,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:27,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:27,958 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:27,958 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:27,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,048 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=979, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/3109571aeb0249ed836febe6a63ce560 2023-04-05 09:54:28,058 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,061 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,064 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,064 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,064 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,064 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,065 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,065 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,066 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,066 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,067 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a710af6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,069 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,069 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,070 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,070 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,070 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,074 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,074 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,074 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,074 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,074 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,078 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,079 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,081 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,081 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,181 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,185 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,186 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,186 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,190 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,190 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,190 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,190 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,192 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1381724d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,199 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,202 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,202 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,204 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,208 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,209 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,209 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,309 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,315 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,315 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,315 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,315 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,318 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,318 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,320 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,321 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,321 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,321 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5a18d0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,327 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,329 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,332 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,332 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,332 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,332 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,332 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,341 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,342 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:28,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,342 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688528433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688528434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:28,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688528436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:28,442 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688528439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688528443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,445 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,446 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,446 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,449 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,449 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,449 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,449 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,450 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c9e837[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,456 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/6c9b777119244f39989260893e981879 2023-04-05 09:54:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,459 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,460 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,461 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,461 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,461 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,467 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,468 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,468 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,467 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,468 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:28,568 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,571 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0d6362[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,585 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,585 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,590 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,595 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,595 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,595 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,595 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,595 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,695 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,698 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,700 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,702 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,702 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,702 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,702 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42866174[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,709 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,709 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,720 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,720 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,720 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,820 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,824 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:28,828 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f07d6c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,835 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,835 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,835 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,837 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,838 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,838 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:28,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,843 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,843 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,843 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,844 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/fc5fb6e972464c45853b804bf511fa6c 2023-04-05 09:54:28,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/3109571aeb0249ed836febe6a63ce560 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560 2023-04-05 09:54:28,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560, entries=150, sequenceid=979, filesize=25.4 K 2023-04-05 09:54:28,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/6c9b777119244f39989260893e981879 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879 2023-04-05 09:54:28,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879, entries=150, sequenceid=979, filesize=11.9 K 2023-04-05 09:54:28,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/fc5fb6e972464c45853b804bf511fa6c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c 2023-04-05 09:54:28,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c, entries=150, sequenceid=979, filesize=11.9 K 2023-04-05 09:54:28,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~523.64 KB/536208, currentSize=0 B/0 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1273ms, sequenceid=979, compaction requested=true 2023-04-05 09:54:28,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:28,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:28,898 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 09:54:28,898 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 09:54:28,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:28,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:28,902 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 135146 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 09:54:28,903 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 63404 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 09:54:28,903 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A is initiating minor compaction (all files) 2023-04-05 09:54:28,903 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B is initiating minor compaction (all files) 2023-04-05 09:54:28,903 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,903 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,903 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/beed3c0399a64803be37f68a2c0b34a6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=132.0 K 2023-04-05 09:54:28,903 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/c41fc1b626f141c69dd62f4afc74eadc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=61.9 K 2023-04-05 09:54:28,903 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting beed3c0399a64803be37f68a2c0b34a6, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462771 2023-04-05 09:54:28,904 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c41fc1b626f141c69dd62f4afc74eadc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462771 2023-04-05 09:54:28,904 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 52f382577aaf416fa2ebab3a50010759, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680688462852 2023-04-05 09:54:28,904 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 963d048ad2084b1bbeb0b21b0551036d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680688462852 2023-04-05 09:54:28,904 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6ed38db042be492aa7f8b2d29a4985a8, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688464935 2023-04-05 09:54:28,905 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 07765b33f3c4441db3e92309c7aed4f9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688464967 2023-04-05 09:54:28,905 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting af272f0237874069ba2947635e846307, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680688465177 2023-04-05 09:54:28,905 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5e086f458dee46659375fa07e5403511, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680688465177 2023-04-05 09:54:28,905 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 12dd493b10d04b5bbe7375e62e0ed32c, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680688465714 2023-04-05 09:54:28,906 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 3109571aeb0249ed836febe6a63ce560, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:28,906 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e7ca8f19ee4a47cfbb6e3b1f73068e49, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680688465714 2023-04-05 09:54:28,907 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6c9b777119244f39989260893e981879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:28,942 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#B#compaction#172 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:28,944 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:28,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:28,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:28,949 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:28,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:28,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:28,950 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,950 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,951 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:28,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:28,953 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#A#compaction#171 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:28,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:28,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:28,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:28,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f8fa015[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:28,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:28,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:28,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:28,989 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:28,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:28,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:28,992 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:28,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:28,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:28,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:28,992 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:28,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,994 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:28,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:28,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:28,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,000 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,000 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,100 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,112 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,128 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,130 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea8def5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,133 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,136 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,139 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,139 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,141 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,145 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:29,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,147 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,147 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,247 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,251 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,251 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,251 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,251 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,257 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,259 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,259 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,259 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,261 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,261 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd48f95[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,265 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,266 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,267 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,270 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,270 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,273 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,273 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,273 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,274 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,277 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,278 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,278 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,378 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,382 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,414 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,416 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,416 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,418 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,420 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/fd1aa04c9e324ae082ef1c4f69c02396 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/fd1aa04c9e324ae082ef1c4f69c02396 2023-04-05 09:54:29,422 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34648a1e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,427 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into fd1aa04c9e324ae082ef1c4f69c02396(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:29,427 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,427 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, priority=10, startTime=1680688468898; duration=0sec 2023-04-05 09:54:29,428 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:29,428 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 09:54:29,428 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,429 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,430 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,430 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,430 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 63404 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 09:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,431 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C is initiating minor compaction (all files) 2023-04-05 09:54:29,431 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,431 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/722955dd6a994b6da587ebfd53b498aa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=61.9 K 2023-04-05 09:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,432 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 722955dd6a994b6da587ebfd53b498aa, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680688462771 2023-04-05 09:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,432 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 19fc394cfa1d4baaae58a9e2a5f8c3cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680688462852 2023-04-05 09:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,433 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 599cbd43876f4b72b5fc45a4fa911ed2, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688464967 2023-04-05 09:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,433 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6d8031a42bac4148acf8fb128e2bec5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680688465177 2023-04-05 09:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,434 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 55e083563ea44b5ab34e15ac10871687, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680688465714 2023-04-05 09:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,434 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting fc5fb6e972464c45853b804bf511fa6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:29,434 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,434 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,441 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,441 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,442 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,442 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,442 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,450 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#C#compaction#173 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:29,472 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/8f92af2d8b85404ebdf9a2f38fe76e9e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8f92af2d8b85404ebdf9a2f38fe76e9e 2023-04-05 09:54:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:29,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:29,478 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 8f92af2d8b85404ebdf9a2f38fe76e9e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:29,478 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,478 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, priority=10, startTime=1680688468898; duration=0sec 2023-04-05 09:54:29,478 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:29,497 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d4a96398a6d444f99d0fb9621034d7e6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4a96398a6d444f99d0fb9621034d7e6 2023-04-05 09:54:29,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1680688469477, max timestamp=1680688469517Num uniques 859; 2023-04-05 09:54:29,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1680688469477, max timestamp=1680688469517Num uniques 859; 2023-04-05 09:54:29,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1680688469477, max timestamp=1680688469517Num uniques 859; 2023-04-05 09:54:29,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 09:54:29,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 09:54:29,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 09:54:29,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688529532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688529533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688529533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688529536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688529537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,542 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,546 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,564 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,564 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,564 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:29,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,566 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16491b8f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,573 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,573 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,577 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,577 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,584 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,586 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,587 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,587 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,587 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688529638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688529641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688529641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688529642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688529642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,687 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,690 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,691 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,691 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,692 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,694 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,694 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,694 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:29,694 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,696 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebe30c5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,712 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,713 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,713 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,722 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,722 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,822 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,827 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,828 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/88a72699631b489cbcfc21812ca0bb87 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/88a72699631b489cbcfc21812ca0bb87 2023-04-05 09:54:29,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,832 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,837 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 88a72699631b489cbcfc21812ca0bb87(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:29,837 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:29,837 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, priority=10, startTime=1680688468898; duration=0sec 2023-04-05 09:54:29,838 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,841 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,841 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,841 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:29,841 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ada8a0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,845 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,845 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688529841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,846 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688529845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688529845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688529847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688529847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,850 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,850 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,858 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,861 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,862 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,862 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,862 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,914 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=994, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/2ff5854d8c8042759d31124081e51839 2023-04-05 09:54:29,962 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:29,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:29,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:29,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:29,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:29,965 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:29,965 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,965 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:29,965 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:29,966 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,966 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,967 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:29,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:29,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:29,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:29,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:29,968 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:29,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:29,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:29,969 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:29,970 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,970 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1509e671[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:29,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:29,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:29,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:29,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:29,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:29,975 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:29,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:29,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:29,978 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:29,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:29,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:29,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:29,978 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:29,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:29,984 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:29,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:29,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:29,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:29,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:29,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:29,985 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:29,985 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:29,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,086 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,089 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,092 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,095 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc2023c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,099 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,106 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,106 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,112 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,112 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,113 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,113 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688530148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688530151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688530152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688530153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688530151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,213 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,216 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,217 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,218 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,225 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,225 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,225 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,225 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,227 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b53b99[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,234 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,234 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,237 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,241 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,242 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,242 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/0664a59b44384a239f87a192d578192e 2023-04-05 09:54:30,342 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,352 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,352 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,352 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,352 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,359 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,359 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,380 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,380 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,380 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,380 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,382 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c356b07[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,386 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,388 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,391 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,391 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,393 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,394 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,396 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,396 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,397 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,397 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,397 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,497 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,500 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d5229e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,509 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,509 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,510 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,511 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,514 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,514 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,525 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,525 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,525 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,525 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,625 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,629 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,634 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddd159a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,637 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,646 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,646 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,648 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,648 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,648 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,648 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,652 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,652 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,653 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,652 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,653 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,653 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688530655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688530655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688530657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688530658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:30,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688530659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,753 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,756 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,757 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,757 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,758 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,760 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,760 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,760 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:30,760 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:30,763 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:30,764 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4099924b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:30,764 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,766 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:30,766 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,766 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:30,766 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:30,766 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:30,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:30,770 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:30,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:30,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:30,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:30,770 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:30,771 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,771 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,771 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:30,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:30,772 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/4fb08a8bca7e437ca8792c5c35b42575 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:30,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:30,777 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:30,777 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:30,777 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:30,777 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:30,777 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:30,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/2ff5854d8c8042759d31124081e51839 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839 2023-04-05 09:54:30,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839, entries=350, sequenceid=994, filesize=52.9 K 2023-04-05 09:54:30,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/0664a59b44384a239f87a192d578192e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e 2023-04-05 09:54:30,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e, entries=100, sequenceid=994, filesize=9.5 K 2023-04-05 09:54:30,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/4fb08a8bca7e437ca8792c5c35b42575 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575 2023-04-05 09:54:30,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575, entries=100, sequenceid=994, filesize=9.5 K 2023-04-05 09:54:30,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1316ms, sequenceid=994, compaction requested=false 2023-04-05 09:54:30,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:30,877 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:30,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:30,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:30,880 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:30,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:30,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:30,881 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,881 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:30,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:30,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,883 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:30,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:30,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:30,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:30,884 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:30,884 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:30,884 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:30,884 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:30,884 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:30,884 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:30,884 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:31,300 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e305940d611c44d2b041377510da2d3e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e305940d611c44d2b041377510da2d3e 2023-04-05 09:54:31,307 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1022, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/c0ef68d45dbb4e32a3760286d5461e8b 2023-04-05 09:54:31,317 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/2e178b62975341b4be892391bb57614d 2023-04-05 09:54:31,329 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/388d709b0b59496bb2a1cc3f4327e294 2023-04-05 09:54:31,334 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/c0ef68d45dbb4e32a3760286d5461e8b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b 2023-04-05 09:54:31,337 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b, entries=150, sequenceid=1022, filesize=25.4 K 2023-04-05 09:54:31,338 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/2e178b62975341b4be892391bb57614d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d 2023-04-05 09:54:31,342 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d, entries=150, sequenceid=1022, filesize=11.9 K 2023-04-05 09:54:31,343 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/388d709b0b59496bb2a1cc3f4327e294 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294 2023-04-05 09:54:31,358 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294, entries=150, sequenceid=1022, filesize=11.9 K 2023-04-05 09:54:31,359 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=0 B/0 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 475ms, sequenceid=1022, compaction requested=true 2023-04-05 09:54:31,359 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:31,359 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:31,361 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3b40c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:31,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:31,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:31,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:31,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:31,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:31,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:31,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,366 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:31,369 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:31,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:31,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:31,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:31,370 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:31,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,379 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,379 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,379 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,379 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,379 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:31,380 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,380 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:31,380 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,380 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:31,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,380 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,380 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:31,380 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:31,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,481 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:31,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:31,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:31,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:31,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:31,486 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:31,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:31,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:31,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,488 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:31,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:31,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:31,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:31,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:31,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:31,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,494 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18edb7b6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:31,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:31,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,498 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:31,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:31,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:31,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:31,503 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:31,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:31,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:31,504 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:31,504 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:31,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,507 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,509 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:31,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,513 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:31,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,513 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:31,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,613 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:31,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:31,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:31,617 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:31,617 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,617 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:31,617 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:31,617 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,618 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,619 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,619 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,619 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,619 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:31,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:31,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:31,624 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:31,624 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:31,624 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,624 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:31,624 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:31,624 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:31,627 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:31,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:31,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9900f4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:31,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:31,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:31,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:31,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:31,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:31,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,633 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:31,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:31,636 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:31,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:31,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:31,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:31,636 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:31,637 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,638 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,638 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:31,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:31,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,644 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,644 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,644 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:31,644 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,644 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,644 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,644 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:31,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:31,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:31,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:31,745 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:31,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:31,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:31,753 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:31,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:31,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:31,755 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,755 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,758 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:31,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:31,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:31,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,760 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:31,761 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:31,761 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:31,761 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,761 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:31,762 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:31,762 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:31,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:31,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:31,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:31,763 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:31,765 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:31,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119ec2c6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:31,765 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:31,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:31,770 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688531769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688531770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688471753, max timestamp=1680688471771Num uniques 859; 2023-04-05 09:54:31,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:54:31,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688471753, max timestamp=1680688471771Num uniques 859; 2023-04-05 09:54:31,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:54:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680688471753, max timestamp=1680688471771Num uniques 859; 2023-04-05 09:54:31,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 09:54:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:31,775 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:31,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:31,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:31,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:31,775 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:31,778 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,778 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,778 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,779 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,780 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:31,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:31,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,783 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:31,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,783 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:31,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688531840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688531841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688531841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688531874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688531874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,883 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:31,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:31,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:31,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:31,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:31,887 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:31,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:31,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:31,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,892 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,892 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,893 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,893 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:31,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:31,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:31,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:31,896 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:31,896 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:31,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:31,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:31,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:31,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:31,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:31,900 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:31,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b97072d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:31,902 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:31,902 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,903 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:31,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:31,905 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:31,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:31,911 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:31,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:31,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:31,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:31,911 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:31,914 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,914 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,914 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:31,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:31,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:31,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:31,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:31,932 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:31,932 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:31,932 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:31,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:31,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:31,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688531944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688531946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:31,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688531947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,032 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:32,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:32,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:32,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:32,035 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:32,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:32,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:32,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:32,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:32,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:32,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:32,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:32,040 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,040 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:32,040 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:32,040 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:32,042 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb251ee[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:32,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:32,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:32,046 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:32,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:32,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:32,051 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:32,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:32,059 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:32,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:32,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:32,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:32,059 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:32,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:32,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:32,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:32,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:32,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:32,065 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:32,065 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:32,065 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:32,065 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:32,065 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:32,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:32,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:32,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688532083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688532083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688532150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688532151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688532151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,166 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:32,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:32,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e32eff6d61c348c1babd7c849b314386 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e32eff6d61c348c1babd7c849b314386 2023-04-05 09:54:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:32,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:32,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:32,176 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:32,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:32,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:32,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:32,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:32,179 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,179 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:32,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:32,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:32,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:32,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:32,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:32,196 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,196 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:32,196 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:32,196 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:32,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:32,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,203 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,203 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:32,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf9606c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:32,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:32,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,207 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:32,207 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,207 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:32,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,211 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1035, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/7257b42416994bccb503fcfdaae3efa2 2023-04-05 09:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:32,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:32,218 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:32,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:32,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:32,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:32,218 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:32,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:32,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:32,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:32,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:32,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:32,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:32,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:32,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:32,233 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:32,233 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:32,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:32,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:32,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1035 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/5ed4159e5eab40e385eca99b24c54c79 2023-04-05 09:54:32,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1035 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/6e924c8195b8436ea311651a74aac97a 2023-04-05 09:54:32,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/7257b42416994bccb503fcfdaae3efa2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2 2023-04-05 09:54:32,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2, entries=400, sequenceid=1035, filesize=59.8 K 2023-04-05 09:54:32,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/5ed4159e5eab40e385eca99b24c54c79 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79 2023-04-05 09:54:32,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79, entries=150, sequenceid=1035, filesize=11.9 K 2023-04-05 09:54:32,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/6e924c8195b8436ea311651a74aac97a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a 2023-04-05 09:54:32,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a, entries=150, sequenceid=1035, filesize=11.9 K 2023-04-05 09:54:32,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 579ms, sequenceid=1035, compaction requested=true 2023-04-05 09:54:32,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:32,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:32,284 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:32,286 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:32,287 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 168170 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:32,287 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A is initiating minor compaction (all files) 2023-04-05 09:54:32,287 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,288 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/88a72699631b489cbcfc21812ca0bb87, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=164.2 K 2023-04-05 09:54:32,288 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 88a72699631b489cbcfc21812ca0bb87, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:32,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:32,290 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:32,290 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B is initiating minor compaction (all files) 2023-04-05 09:54:32,290 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,290 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/fd1aa04c9e324ae082ef1c4f69c02396, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=45.8 K 2023-04-05 09:54:32,289 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2ff5854d8c8042759d31124081e51839, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680688469454 2023-04-05 09:54:32,291 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting fd1aa04c9e324ae082ef1c4f69c02396, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:32,291 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c0ef68d45dbb4e32a3760286d5461e8b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1680688469517 2023-04-05 09:54:32,291 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:32,291 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0664a59b44384a239f87a192d578192e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680688469474 2023-04-05 09:54:32,292 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7257b42416994bccb503fcfdaae3efa2, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680688471672 2023-04-05 09:54:32,292 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2e178b62975341b4be892391bb57614d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1680688469517 2023-04-05 09:54:32,292 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5ed4159e5eab40e385eca99b24c54c79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680688471675 2023-04-05 09:54:32,318 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#B#compaction#184 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:32,328 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#A#compaction#183 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:32,334 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:32,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:32,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:32,353 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:32,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:32,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:32,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,355 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:32,355 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:32,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:32,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:32,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:32,364 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:32,365 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:32,365 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:32,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:32,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688532407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688532410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688532454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688532455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688532457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688532512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688532519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688532715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688532726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,751 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/0d32d6cf2ede4e2fb74a8065f4efc767 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0d32d6cf2ede4e2fb74a8065f4efc767 2023-04-05 09:54:32,761 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/B of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 0d32d6cf2ede4e2fb74a8065f4efc767(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:32,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:32,761 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, priority=12, startTime=1680688472285; duration=0sec 2023-04-05 09:54:32,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:32,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:32,764 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:32,764 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C is initiating minor compaction (all files) 2023-04-05 09:54:32,764 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C in TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:32,764 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8f92af2d8b85404ebdf9a2f38fe76e9e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp, totalSize=45.8 K 2023-04-05 09:54:32,766 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8f92af2d8b85404ebdf9a2f38fe76e9e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680688467285 2023-04-05 09:54:32,766 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4fb08a8bca7e437ca8792c5c35b42575, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680688469474 2023-04-05 09:54:32,766 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 388d709b0b59496bb2a1cc3f4327e294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1680688469517 2023-04-05 09:54:32,767 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6e924c8195b8436ea311651a74aac97a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680688471675 2023-04-05 09:54:32,778 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 3f59f85b0ea42d0cfdd0e5ffae0c1e23#C#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:54:32,790 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040559fde3f13c014de9b1c54852b896acae to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040559fde3f13c014de9b1c54852b896acae 2023-04-05 09:54:32,794 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/f5d57bab45c44d26844e8247973424b6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/f5d57bab45c44d26844e8247973424b6 2023-04-05 09:54:32,798 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1069, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/484f1411bf834d3aa9ebf3ac6e9ff03a 2023-04-05 09:54:32,807 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/C of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into f5d57bab45c44d26844e8247973424b6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:32,807 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:32,807 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, priority=12, startTime=1680688472291; duration=0sec 2023-04-05 09:54:32,807 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:32,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688532960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688532964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:32,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688532965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688533019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688533029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,177 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/60dfccb821164254937453f5f2451cca as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/60dfccb821164254937453f5f2451cca 2023-04-05 09:54:33,182 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3f59f85b0ea42d0cfdd0e5ffae0c1e23/A of 3f59f85b0ea42d0cfdd0e5ffae0c1e23 into 60dfccb821164254937453f5f2451cca(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:33,182 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:33,182 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23., storeName=3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, priority=12, startTime=1680688472284; duration=0sec 2023-04-05 09:54:33,182 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:33,220 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1069 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/6a2af329c38f47f589a115a4fd68b1d1 2023-04-05 09:54:33,230 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1069 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/869819a672fa42929eaa8d6a59deb19c 2023-04-05 09:54:33,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/484f1411bf834d3aa9ebf3ac6e9ff03a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/484f1411bf834d3aa9ebf3ac6e9ff03a 2023-04-05 09:54:33,238 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/484f1411bf834d3aa9ebf3ac6e9ff03a, entries=150, sequenceid=1069, filesize=25.4 K 2023-04-05 09:54:33,239 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/6a2af329c38f47f589a115a4fd68b1d1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6a2af329c38f47f589a115a4fd68b1d1 2023-04-05 09:54:33,243 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6a2af329c38f47f589a115a4fd68b1d1, entries=150, sequenceid=1069, filesize=11.9 K 2023-04-05 09:54:33,244 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/869819a672fa42929eaa8d6a59deb19c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/869819a672fa42929eaa8d6a59deb19c 2023-04-05 09:54:33,254 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/869819a672fa42929eaa8d6a59deb19c, entries=150, sequenceid=1069, filesize=11.9 K 2023-04-05 09:54:33,255 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 890ms, sequenceid=1069, compaction requested=false 2023-04-05 09:54:33,255 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:33,255 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:33,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,257 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:33,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:33,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c960f27[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:33,259 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:33,259 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,263 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:33,263 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:33,265 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:33,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:33,268 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:33,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:33,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:33,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:33,268 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:33,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:33,270 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:33,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:33,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:33,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,272 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:33,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:33,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:33,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:33,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:33,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:33,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:33,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:33,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:33,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,275 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:33,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:33,275 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:33,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:33,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,375 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:33,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:33,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:33,378 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:33,378 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:33,378 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:33,379 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:33,379 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,379 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:33,381 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:33,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:33,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,382 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:33,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:33,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:33,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:33,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:33,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:33,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:33,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:33,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:33,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:33,383 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:33,383 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. started... 2023-04-05 09:54:33,383 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:33,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:33,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. as already flushing 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688473549, max timestamp=1680688473575Num uniques 859; 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688473549, max timestamp=1680688473575Num uniques 859; 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688473549, max timestamp=1680688473575Num uniques 859; 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:33,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:33,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688533619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688533619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688533724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688533724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,728 DEBUG [Thread-904] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:62753 2023-04-05 09:54:33,728 DEBUG [Thread-904] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:33,729 DEBUG [Thread-910] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:62753 2023-04-05 09:54:33,729 DEBUG [Thread-910] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:33,730 DEBUG [Thread-908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:62753 2023-04-05 09:54:33,730 DEBUG [Thread-908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:33,731 DEBUG [Thread-902] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:62753 2023-04-05 09:54:33,731 DEBUG [Thread-902] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:33,798 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f0c079c05abf416984aba103c9ecf226 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f0c079c05abf416984aba103c9ecf226 2023-04-05 09:54:33,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688533929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688533931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51612 deadline: 1680688533968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51608 deadline: 1680688533972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:33,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:33,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51628 deadline: 1680688533975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:34,203 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1083, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/824a7ad0dc784a97ad153ef678232be6 2023-04-05 09:54:34,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688534232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:34,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688534233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:34,612 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/383f1ca64d9e4572ae933dd99c327d01 2023-04-05 09:54:34,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51636 deadline: 1680688534734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:34,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51616 deadline: 1680688534738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,022 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/8d132b7ec2274d9f91384e6c594091c6 2023-04-05 09:54:35,026 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/824a7ad0dc784a97ad153ef678232be6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/824a7ad0dc784a97ad153ef678232be6 2023-04-05 09:54:35,030 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/824a7ad0dc784a97ad153ef678232be6, entries=150, sequenceid=1083, filesize=25.4 K 2023-04-05 09:54:35,030 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/383f1ca64d9e4572ae933dd99c327d01 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/383f1ca64d9e4572ae933dd99c327d01 2023-04-05 09:54:35,034 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/383f1ca64d9e4572ae933dd99c327d01, entries=150, sequenceid=1083, filesize=11.9 K 2023-04-05 09:54:35,035 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/8d132b7ec2274d9f91384e6c594091c6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8d132b7ec2274d9f91384e6c594091c6 2023-04-05 09:54:35,038 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8d132b7ec2274d9f91384e6c594091c6, entries=150, sequenceid=1083, filesize=11.9 K 2023-04-05 09:54:35,039 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1656ms, sequenceid=1083, compaction requested=true 2023-04-05 09:54:35,039 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:35,039 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8bc7f7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:35,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:35,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:35,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:35,050 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:35,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:35,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:35,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:35,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:35,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:35,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:35,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:35,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:35,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:35,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:35,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:35,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:35,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:35,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:35,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:35,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:35,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:35,057 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:35,057 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:35,057 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:35,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:35,057 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:35,057 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,057 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:35,057 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:35,157 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:35,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:35,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:35,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:54:35,741 DEBUG [Thread-895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:62753 2023-04-05 09:54:35,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:35,741 DEBUG [Thread-895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:35,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:35,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:35,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:35,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:35,744 DEBUG [Thread-897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:62753 2023-04-05 09:54:35,744 DEBUG [Thread-897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,975 DEBUG [Thread-893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bbdba17 to 127.0.0.1:62753 2023-04-05 09:54:35,975 DEBUG [Thread-893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,980 DEBUG [Thread-899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:62753 2023-04-05 09:54:35,980 DEBUG [Thread-899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,992 DEBUG [Thread-891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29f78b2c to 127.0.0.1:62753 2023-04-05 09:54:35,993 DEBUG [Thread-891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 205 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 237 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 237 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 195 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5164 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4877 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2080 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6240 rows 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2070 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6210 rows 2023-04-05 09:54:35,993 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:54:35,993 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31436cf6 to 127.0.0.1:62753 2023-04-05 09:54:35,993 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:54:35,995 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:54:35,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:54:35,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:35,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:36,000 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688475999"}]},"ts":"1680688475999"} 2023-04-05 09:54:36,001 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:54:36,002 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:54:36,003 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, UNASSIGN}] 2023-04-05 09:54:36,004 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, UNASSIGN 2023-04-05 09:54:36,004 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:36,004 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688476004"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688476004"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688476004"}]},"ts":"1680688476004"} 2023-04-05 09:54:36,005 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:36,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:36,155 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cb9ba848487448b5818910b7137f855c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb9ba848487448b5818910b7137f855c 2023-04-05 09:54:36,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:36,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3f59f85b0ea42d0cfdd0e5ffae0c1e23, disabling compactions & flushes 2023-04-05 09:54:36,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:36,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:36,560 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1120, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/46733ad3c63a4b579b285b4b64e03438 2023-04-05 09:54:36,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:36,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1120 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/728b728dbfc94629b5cbaebb375e8c1d 2023-04-05 09:54:37,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:37,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1120 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/8438a4312b12476f8e0eae7f14029839 2023-04-05 09:54:37,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/46733ad3c63a4b579b285b4b64e03438 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/46733ad3c63a4b579b285b4b64e03438 2023-04-05 09:54:37,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/46733ad3c63a4b579b285b4b64e03438, entries=150, sequenceid=1120, filesize=25.4 K 2023-04-05 09:54:37,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/728b728dbfc94629b5cbaebb375e8c1d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/728b728dbfc94629b5cbaebb375e8c1d 2023-04-05 09:54:37,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/728b728dbfc94629b5cbaebb375e8c1d, entries=150, sequenceid=1120, filesize=11.9 K 2023-04-05 09:54:37,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/8438a4312b12476f8e0eae7f14029839 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8438a4312b12476f8e0eae7f14029839 2023-04-05 09:54:37,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8438a4312b12476f8e0eae7f14029839, entries=150, sequenceid=1120, filesize=11.9 K 2023-04-05 09:54:37,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=26.84 KB/27480 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1662ms, sequenceid=1120, compaction requested=true 2023-04-05 09:54:37,403 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:37,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:37,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:37,403 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. because compaction request was cancelled 2023-04-05 09:54:37,403 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. because compaction request was cancelled 2023-04-05 09:54:37,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:37,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:37,403 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. because compaction request was cancelled 2023-04-05 09:54:37,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. after waiting 0 ms 2023-04-05 09:54:37,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:37,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:37,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3f59f85b0ea42d0cfdd0e5ffae0c1e23 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=A 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=B 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3f59f85b0ea42d0cfdd0e5ffae0c1e23, store=C 2023-04-05 09:54:37,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:37,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051946bb5ca9e143ce857895ea6e463b0e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051946bb5ca9e143ce857895ea6e463b0e 2023-04-05 09:54:38,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:38,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1127, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/b41a2d9ecdfd4a10b69185fb2749f0fb 2023-04-05 09:54:38,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1127 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/ff20fe92c23040fba6d1c62764d29ed2 2023-04-05 09:54:39,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1127 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/81b2625ba18d437ea38de90d50838a8b 2023-04-05 09:54:39,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/A/b41a2d9ecdfd4a10b69185fb2749f0fb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/b41a2d9ecdfd4a10b69185fb2749f0fb 2023-04-05 09:54:39,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/b41a2d9ecdfd4a10b69185fb2749f0fb, entries=150, sequenceid=1127, filesize=25.4 K 2023-04-05 09:54:39,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/B/ff20fe92c23040fba6d1c62764d29ed2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ff20fe92c23040fba6d1c62764d29ed2 2023-04-05 09:54:39,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ff20fe92c23040fba6d1c62764d29ed2, entries=150, sequenceid=1127, filesize=11.9 K 2023-04-05 09:54:39,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/.tmp/C/81b2625ba18d437ea38de90d50838a8b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/81b2625ba18d437ea38de90d50838a8b 2023-04-05 09:54:39,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/81b2625ba18d437ea38de90d50838a8b, entries=150, sequenceid=1127, filesize=11.9 K 2023-04-05 09:54:39,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 3f59f85b0ea42d0cfdd0e5ffae0c1e23 in 1667ms, sequenceid=1127, compaction requested=true 2023-04-05 09:54:39,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2430abc762f04834a481ccc1574242aa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c257e24a5e1745e6b970214af8eb2d12, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/beed3c0399a64803be37f68a2c0b34a6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/88a72699631b489cbcfc21812ca0bb87, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2] to archive 2023-04-05 09:54:39,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:39,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/8acb2090bf3e4bfbbdca2ab3cb9f7e3f 2023-04-05 09:54:39,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/681208158d5a42a28d52d860ffc9032b 2023-04-05 09:54:39,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/e72849dc58bf44acbae498b060f5c149 2023-04-05 09:54:39,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/4a2fb4cb0dec42599b4a1a1927340d66 2023-04-05 09:54:39,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/20136159529e4ab6bcf324900aeddb3a 2023-04-05 09:54:39,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2430abc762f04834a481ccc1574242aa to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2430abc762f04834a481ccc1574242aa 2023-04-05 09:54:39,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7d38de43378c4b8aab24845db98f4e1d 2023-04-05 09:54:39,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/a7b79184a523483992d83dc884c87602 2023-04-05 09:54:39,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/597fcab4eb534d6f85350166a778ee56 2023-04-05 09:54:39,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c257e24a5e1745e6b970214af8eb2d12 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c257e24a5e1745e6b970214af8eb2d12 2023-04-05 09:54:39,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/d39b25b1131b4890803e75288e6bb712 2023-04-05 09:54:39,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af34b05bcc954c219758d46ac8148541 2023-04-05 09:54:39,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/beed3c0399a64803be37f68a2c0b34a6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/beed3c0399a64803be37f68a2c0b34a6 2023-04-05 09:54:39,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/52f382577aaf416fa2ebab3a50010759 2023-04-05 09:54:39,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/6ed38db042be492aa7f8b2d29a4985a8 2023-04-05 09:54:39,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/af272f0237874069ba2947635e846307 2023-04-05 09:54:39,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/12dd493b10d04b5bbe7375e62e0ed32c 2023-04-05 09:54:39,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/88a72699631b489cbcfc21812ca0bb87 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/88a72699631b489cbcfc21812ca0bb87 2023-04-05 09:54:39,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/3109571aeb0249ed836febe6a63ce560 2023-04-05 09:54:39,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/2ff5854d8c8042759d31124081e51839 2023-04-05 09:54:39,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/c0ef68d45dbb4e32a3760286d5461e8b 2023-04-05 09:54:39,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/7257b42416994bccb503fcfdaae3efa2 2023-04-05 09:54:39,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/b1fa6e9c14174a27b31d2e2bcb4efbba, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2dc2a6982bca4d30a9aeed546603d3e9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/c41fc1b626f141c69dd62f4afc74eadc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/fd1aa04c9e324ae082ef1c4f69c02396, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79] to archive 2023-04-05 09:54:39,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:39,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3fd3f1dba884b999a5b419efa55167c 2023-04-05 09:54:39,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/8a2569048cb2498a8669f2e8579cd9ce 2023-04-05 09:54:39,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/42de927a7673410fa9ae5fb1b0cc4ba6 2023-04-05 09:54:39,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e8f5fee29b2345d4b342e52585140b58 2023-04-05 09:54:39,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/b1fa6e9c14174a27b31d2e2bcb4efbba to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/b1fa6e9c14174a27b31d2e2bcb4efbba 2023-04-05 09:54:39,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/96efe58dfab14deb8de3a70e32729957 2023-04-05 09:54:39,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ef523c3be31b485d8988861a86569339 2023-04-05 09:54:39,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/cf399a5dc30f42e9a6d1ed9826321224 2023-04-05 09:54:39,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2dc2a6982bca4d30a9aeed546603d3e9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2dc2a6982bca4d30a9aeed546603d3e9 2023-04-05 09:54:39,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/7512c593b5594382b5b9caf61d45f72f 2023-04-05 09:54:39,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/370ca64f3e4e46598bac10384f768444 2023-04-05 09:54:39,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/c41fc1b626f141c69dd62f4afc74eadc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/c41fc1b626f141c69dd62f4afc74eadc 2023-04-05 09:54:39,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/f3349f4a4f5e4e77b3d1cc8418d2dd56 2023-04-05 09:54:39,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/963d048ad2084b1bbeb0b21b0551036d 2023-04-05 09:54:39,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/07765b33f3c4441db3e92309c7aed4f9 2023-04-05 09:54:39,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5e086f458dee46659375fa07e5403511 2023-04-05 09:54:39,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/e7ca8f19ee4a47cfbb6e3b1f73068e49 2023-04-05 09:54:39,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/fd1aa04c9e324ae082ef1c4f69c02396 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/fd1aa04c9e324ae082ef1c4f69c02396 2023-04-05 09:54:39,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6c9b777119244f39989260893e981879 2023-04-05 09:54:39,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0664a59b44384a239f87a192d578192e 2023-04-05 09:54:39,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/2e178b62975341b4be892391bb57614d 2023-04-05 09:54:39,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/5ed4159e5eab40e385eca99b24c54c79 2023-04-05 09:54:39,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/25a5500cb3194e13b9803c5e10085c3f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/212720d6e30640dca6cce01e5dbe303d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/722955dd6a994b6da587ebfd53b498aa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8f92af2d8b85404ebdf9a2f38fe76e9e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a] to archive 2023-04-05 09:54:39,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:54:39,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/5119d518e0d24406959000839907f897 2023-04-05 09:54:39,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/a2e692a5051a41ebb3c78122e0d3fc2e 2023-04-05 09:54:39,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/aec64b37aa6344d69f39530964b35b69 2023-04-05 09:54:39,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc21a1a307c548588a1f04814f28c760 2023-04-05 09:54:39,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/25a5500cb3194e13b9803c5e10085c3f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/25a5500cb3194e13b9803c5e10085c3f 2023-04-05 09:54:39,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/79d942576f9649f9adc7eeefa2fbbe8a 2023-04-05 09:54:39,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/654b1597938a485282d09e4ec8f88fb8 2023-04-05 09:54:39,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4cfe3150735944269f4ea4f8281fbbcf 2023-04-05 09:54:39,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/212720d6e30640dca6cce01e5dbe303d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/212720d6e30640dca6cce01e5dbe303d 2023-04-05 09:54:39,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/14daa788408f430f9253d3f6e089a213 2023-04-05 09:54:39,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/0f78df38b7a946e190422b98bcd59fc4 2023-04-05 09:54:39,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/722955dd6a994b6da587ebfd53b498aa to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/722955dd6a994b6da587ebfd53b498aa 2023-04-05 09:54:39,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/c66e135ed14f484190a827803d7af1b0 2023-04-05 09:54:39,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/19fc394cfa1d4baaae58a9e2a5f8c3cf 2023-04-05 09:54:39,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/599cbd43876f4b72b5fc45a4fa911ed2 2023-04-05 09:54:39,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6d8031a42bac4148acf8fb128e2bec5b 2023-04-05 09:54:39,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/55e083563ea44b5ab34e15ac10871687 2023-04-05 09:54:39,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8f92af2d8b85404ebdf9a2f38fe76e9e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8f92af2d8b85404ebdf9a2f38fe76e9e 2023-04-05 09:54:39,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/fc5fb6e972464c45853b804bf511fa6c 2023-04-05 09:54:39,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/4fb08a8bca7e437ca8792c5c35b42575 2023-04-05 09:54:39,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/388d709b0b59496bb2a1cc3f4327e294 2023-04-05 09:54:39,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/6e924c8195b8436ea311651a74aac97a 2023-04-05 09:54:39,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits/1130.seqid, newMaxSeqId=1130, maxSeqId=4 2023-04-05 09:54:39,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23. 2023-04-05 09:54:39,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3f59f85b0ea42d0cfdd0e5ffae0c1e23: 2023-04-05 09:54:39,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:39,180 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3f59f85b0ea42d0cfdd0e5ffae0c1e23, regionState=CLOSED 2023-04-05 09:54:39,180 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688479180"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688479180"}]},"ts":"1680688479180"} 2023-04-05 09:54:39,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-05 09:54:39,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 3f59f85b0ea42d0cfdd0e5ffae0c1e23, server=jenkins-hbase4.apache.org,46827,1680688416991 in 3.1760 sec 2023-04-05 09:54:39,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-05 09:54:39,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3f59f85b0ea42d0cfdd0e5ffae0c1e23, UNASSIGN in 3.1790 sec 2023-04-05 09:54:39,184 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688479184"}]},"ts":"1680688479184"} 2023-04-05 09:54:39,184 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:54:39,186 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:54:39,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.1910 sec 2023-04-05 09:54:40,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 09:54:40,104 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-05 09:54:40,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:54:40,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,108 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 09:54:40,109 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,115 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:40,118 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits] 2023-04-05 09:54:40,122 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/46733ad3c63a4b579b285b4b64e03438 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/46733ad3c63a4b579b285b4b64e03438 2023-04-05 09:54:40,124 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/484f1411bf834d3aa9ebf3ac6e9ff03a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/484f1411bf834d3aa9ebf3ac6e9ff03a 2023-04-05 09:54:40,125 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/60dfccb821164254937453f5f2451cca to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/60dfccb821164254937453f5f2451cca 2023-04-05 09:54:40,127 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/824a7ad0dc784a97ad153ef678232be6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/824a7ad0dc784a97ad153ef678232be6 2023-04-05 09:54:40,129 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/b41a2d9ecdfd4a10b69185fb2749f0fb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/A/b41a2d9ecdfd4a10b69185fb2749f0fb 2023-04-05 09:54:40,133 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0d32d6cf2ede4e2fb74a8065f4efc767 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/0d32d6cf2ede4e2fb74a8065f4efc767 2023-04-05 09:54:40,139 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/383f1ca64d9e4572ae933dd99c327d01 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/383f1ca64d9e4572ae933dd99c327d01 2023-04-05 09:54:40,141 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6a2af329c38f47f589a115a4fd68b1d1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/6a2af329c38f47f589a115a4fd68b1d1 2023-04-05 09:54:40,143 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/728b728dbfc94629b5cbaebb375e8c1d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/728b728dbfc94629b5cbaebb375e8c1d 2023-04-05 09:54:40,144 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ff20fe92c23040fba6d1c62764d29ed2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/B/ff20fe92c23040fba6d1c62764d29ed2 2023-04-05 09:54:40,147 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/81b2625ba18d437ea38de90d50838a8b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/81b2625ba18d437ea38de90d50838a8b 2023-04-05 09:54:40,148 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8438a4312b12476f8e0eae7f14029839 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8438a4312b12476f8e0eae7f14029839 2023-04-05 09:54:40,149 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/869819a672fa42929eaa8d6a59deb19c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/869819a672fa42929eaa8d6a59deb19c 2023-04-05 09:54:40,150 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8d132b7ec2274d9f91384e6c594091c6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/8d132b7ec2274d9f91384e6c594091c6 2023-04-05 09:54:40,151 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/f5d57bab45c44d26844e8247973424b6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/C/f5d57bab45c44d26844e8247973424b6 2023-04-05 09:54:40,158 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits/1130.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23/recovered.edits/1130.seqid 2023-04-05 09:54:40,159 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/3f59f85b0ea42d0cfdd0e5ffae0c1e23 2023-04-05 09:54:40,159 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:54:40,160 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:54:40,160 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 09:54:40,165 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405085c39238f344b2f925a628995428b6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405085c39238f344b2f925a628995428b6c 2023-04-05 09:54:40,166 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051946bb5ca9e143ce857895ea6e463b0e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051946bb5ca9e143ce857895ea6e463b0e 2023-04-05 09:54:40,168 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052c51af334608473c8412650507be3bef to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052c51af334608473c8412650507be3bef 2023-04-05 09:54:40,169 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040543252eede6844bb28d6997ba2c82e19d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040543252eede6844bb28d6997ba2c82e19d 2023-04-05 09:54:40,177 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054890fd41e993436ea71741f6f6619102 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054890fd41e993436ea71741f6f6619102 2023-04-05 09:54:40,178 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040559fde3f13c014de9b1c54852b896acae to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040559fde3f13c014de9b1c54852b896acae 2023-04-05 09:54:40,179 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055c1fc6818a79485c9cf64f52a035c108 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055c1fc6818a79485c9cf64f52a035c108 2023-04-05 09:54:40,180 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564545ded91584bfdad3251b69989c0b7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564545ded91584bfdad3251b69989c0b7 2023-04-05 09:54:40,182 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056c37920026bc4496a97852337c1c73e8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056c37920026bc4496a97852337c1c73e8 2023-04-05 09:54:40,183 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056ce8f0b08ed74ab6a25de0e47d68bd86 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056ce8f0b08ed74ab6a25de0e47d68bd86 2023-04-05 09:54:40,184 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057f265d4d73e34fd58f8cd3fca2bd3591 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057f265d4d73e34fd58f8cd3fca2bd3591 2023-04-05 09:54:40,185 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058421203888cb47c39897867af51820f0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058421203888cb47c39897867af51820f0 2023-04-05 09:54:40,186 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040590c51e0863624de684836fd27789299c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040590c51e0863624de684836fd27789299c 2023-04-05 09:54:40,187 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c7a0b1f73956475183444a995e25884d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c7a0b1f73956475183444a995e25884d 2023-04-05 09:54:40,188 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb9ba848487448b5818910b7137f855c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb9ba848487448b5818910b7137f855c 2023-04-05 09:54:40,189 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd035bea39c84ec39a9c5b4db82c7363 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd035bea39c84ec39a9c5b4db82c7363 2023-04-05 09:54:40,190 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4a96398a6d444f99d0fb9621034d7e6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4a96398a6d444f99d0fb9621034d7e6 2023-04-05 09:54:40,191 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6105cf0d4a34b45a9cd87e7288c80cb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6105cf0d4a34b45a9cd87e7288c80cb 2023-04-05 09:54:40,193 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e305940d611c44d2b041377510da2d3e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e305940d611c44d2b041377510da2d3e 2023-04-05 09:54:40,194 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e32eff6d61c348c1babd7c849b314386 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e32eff6d61c348c1babd7c849b314386 2023-04-05 09:54:40,195 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f0c079c05abf416984aba103c9ecf226 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f0c079c05abf416984aba103c9ecf226 2023-04-05 09:54:40,196 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2478dc21b974b7cadb52956e9e871f7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2478dc21b974b7cadb52956e9e871f7 2023-04-05 09:54:40,196 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:54:40,198 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,201 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:54:40,202 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:54:40,203 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,203 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:54:40,204 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688480203"}]},"ts":"9223372036854775807"} 2023-04-05 09:54:40,205 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:54:40,205 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3f59f85b0ea42d0cfdd0e5ffae0c1e23, NAME => 'TestAcidGuarantees,,1680688450590.3f59f85b0ea42d0cfdd0e5ffae0c1e23.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:54:40,205 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:54:40,205 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688480205"}]},"ts":"9223372036854775807"} 2023-04-05 09:54:40,207 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:54:40,209 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 09:54:40,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 104 msec 2023-04-05 09:54:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 09:54:40,411 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-05 09:54:40,423 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 230), OpenFileDescriptor=539 (was 563), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=426 (was 363) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1123 (was 1391) 2023-04-05 09:54:40,434 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=426, ProcessCount=169, AvailableMemoryMB=1123 2023-04-05 09:54:40,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:54:40,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:54:40,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:54:40,438 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:54:40,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-05 09:54:40,439 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:54:40,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:40,440 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:54:40,440 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 empty. 2023-04-05 09:54:40,441 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:54:40,441 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:54:40,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:40,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:40,848 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:54:40,850 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8eb0aa9287d767302230619672527872, NAME => 'TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:54:41,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8eb0aa9287d767302230619672527872, disabling compactions & flushes 2023-04-05 09:54:41,256 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. after waiting 0 ms 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,256 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,256 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:41,259 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:54:41,260 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688481260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688481260"}]},"ts":"1680688481260"} 2023-04-05 09:54:41,261 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:54:41,262 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:54:41,262 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688481262"}]},"ts":"1680688481262"} 2023-04-05 09:54:41,263 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:54:41,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, ASSIGN}] 2023-04-05 09:54:41,267 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, ASSIGN 2023-04-05 09:54:41,268 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:54:41,419 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8eb0aa9287d767302230619672527872, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:41,419 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688481419"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688481419"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688481419"}]},"ts":"1680688481419"} 2023-04-05 09:54:41,420 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:54:41,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:41,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8eb0aa9287d767302230619672527872, NAME => 'TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:54:41,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:54:41,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,578 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,580 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A 2023-04-05 09:54:41,580 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A 2023-04-05 09:54:41,580 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:41,581 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8eb0aa9287d767302230619672527872 columnFamilyName A 2023-04-05 09:54:41,581 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(310): Store=8eb0aa9287d767302230619672527872/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:41,581 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,582 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B 2023-04-05 09:54:41,583 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B 2023-04-05 09:54:41,583 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:41,583 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8eb0aa9287d767302230619672527872 columnFamilyName B 2023-04-05 09:54:41,583 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(310): Store=8eb0aa9287d767302230619672527872/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:41,583 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,584 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C 2023-04-05 09:54:41,584 DEBUG [StoreOpener-8eb0aa9287d767302230619672527872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C 2023-04-05 09:54:41,585 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:54:41,585 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8eb0aa9287d767302230619672527872 columnFamilyName C 2023-04-05 09:54:41,585 INFO [StoreOpener-8eb0aa9287d767302230619672527872-1] regionserver.HStore(310): Store=8eb0aa9287d767302230619672527872/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:54:41,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:54:41,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8eb0aa9287d767302230619672527872 2023-04-05 09:54:41,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:54:41,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8eb0aa9287d767302230619672527872; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73967543, jitterRate=0.10220228135585785}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:54:41,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:41,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., pid=30, masterSystemTime=1680688481572 2023-04-05 09:54:41,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:41,593 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8eb0aa9287d767302230619672527872, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:41,593 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688481593"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688481593"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688481593"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688481593"}]},"ts":"1680688481593"} 2023-04-05 09:54:41,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-05 09:54:41,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 in 174 msec 2023-04-05 09:54:41,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-05 09:54:41,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, ASSIGN in 330 msec 2023-04-05 09:54:41,597 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:54:41,597 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688481597"}]},"ts":"1680688481597"} 2023-04-05 09:54:41,598 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:54:41,599 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:54:41,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1640 sec 2023-04-05 09:54:42,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 09:54:42,543 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-05 09:54:42,544 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x22b4eae5 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,550 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12e82edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,551 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,552 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,553 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:54:42,555 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:54:42,556 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,560 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,560 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,564 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,564 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,567 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,567 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,573 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,573 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,576 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,577 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,582 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,582 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,585 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,586 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,591 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,591 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,595 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,595 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:54:42,599 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:54:42,603 DEBUG [hconnection-0x38c5e85e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,604 DEBUG [hconnection-0x483d971b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,604 DEBUG [hconnection-0x1d6eadf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:42,605 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,606 DEBUG [hconnection-0xfe5b0ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,606 DEBUG [hconnection-0x670edb4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,606 DEBUG [hconnection-0x3f46d1c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,606 DEBUG [hconnection-0x55d3ba9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,607 DEBUG [hconnection-0x44a33a9c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,607 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,609 DEBUG [hconnection-0x648f3050-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,611 DEBUG [hconnection-0x662beb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:54:42,613 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,613 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,614 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,614 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,614 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,614 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,615 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,626 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:54:42,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:42,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:42,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:42,634 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:42,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:42,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:42,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:54:42,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,639 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:42,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:42,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:42,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:42,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:42,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:42,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:42,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:42,642 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,642 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:42,642 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:42,642 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:42,651 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:42,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135ac4c9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:42,653 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:42,653 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,654 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:42,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:42,656 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:42,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:42,659 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:42,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:42,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:42,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:42,659 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:42,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,662 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:42,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:42,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,666 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:42,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688542662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,666 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:42,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688542662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688542664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688542665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688542665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,766 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:42,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:42,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688542767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:42,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688542767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688542768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688542769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688542769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:42,781 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:42,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:42,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:42,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:42,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:42,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:42,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:42,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:42,796 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,796 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:42,796 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:42,796 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:42,798 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c03cce[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:42,800 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:42,800 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:42,803 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,803 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:42,804 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:42,807 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:42,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:42,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:42,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:42,807 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:42,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:42,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,814 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:42,814 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:42,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,915 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:42,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:42,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:42,918 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:42,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:42,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:42,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,921 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:42,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:42,922 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,922 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:42,922 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:42,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd3eaa6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:42,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:42,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:42,927 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:42,927 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:42,927 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:42,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:42,930 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:42,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:42,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:42,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:42,931 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:42,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:42,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,936 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,936 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:42,936 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:42,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:42,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:42,936 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:42,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:42,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:42,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:42,937 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:42,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:42,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:42,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:42,937 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688542969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688542971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688542971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688542972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:42,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688542972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,037 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,041 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,045 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,047 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,047 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,047 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,047 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec630d4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,052 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,054 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,056 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,056 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,062 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,062 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,062 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:43,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/dc6beea24f7848e5a29f565f15bee1f9 2023-04-05 09:54:43,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,162 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,165 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,170 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,170 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,170 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,170 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,171 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7fc549[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,173 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,174 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,174 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,174 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,175 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,175 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,175 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,178 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,178 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,181 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,181 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,181 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,182 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,185 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,185 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:43,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,185 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,185 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,185 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688543273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688543274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688543274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688543275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688543276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,285 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,289 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,290 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,292 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,292 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,292 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,292 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192bb2a4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,295 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,302 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,302 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,302 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,302 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,302 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,309 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,309 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,409 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,413 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,415 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,419 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,419 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9bdfdfd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,423 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,428 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,433 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,445 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,445 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/f28dfcccf16446449eb29c76bc5e1a34 2023-04-05 09:54:43,545 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,549 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,549 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,554 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,556 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e23c8e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,563 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,563 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,565 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,565 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,565 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,570 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,570 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,683 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,687 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,691 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,691 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,691 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,691 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,693 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2f696[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,696 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,705 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,705 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,712 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,712 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688543776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688543777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688543779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688543780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688543782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,812 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,815 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,818 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,820 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,820 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,820 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:43,820 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:43,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,823 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d8cad4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:43,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:43,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:43,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:43,827 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,828 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:43,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:43,830 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:43,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:43,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:43,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:43,831 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:43,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:43,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:43,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:43,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:43,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:43,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:43,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:43,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:43,841 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:43,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,841 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:43,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:43,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/59e4679671ff4876ab6d36f77d116f50 2023-04-05 09:54:43,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/dc6beea24f7848e5a29f565f15bee1f9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9 2023-04-05 09:54:43,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 09:54:43,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/f28dfcccf16446449eb29c76bc5e1a34 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34 2023-04-05 09:54:43,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 09:54:43,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/59e4679671ff4876ab6d36f77d116f50 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50 2023-04-05 09:54:43,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 09:54:43,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8eb0aa9287d767302230619672527872 in 1268ms, sequenceid=14, compaction requested=false 2023-04-05 09:54:43,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:43,941 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:43,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:43,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:43,945 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:43,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:43,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:43,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:43,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:43,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:43,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:43,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:43,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:43,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:43,948 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:43,949 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:43,949 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:43,960 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/c9b9e70c7d64444f90dcf145e8e504e6 2023-04-05 09:54:44,432 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5996f4881ea7494da5501c76439d71e1 2023-04-05 09:54:44,720 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:54:44,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:44,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:44,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688544789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688544790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688544793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688544793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688544793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,844 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/7f87be579f414c3b8d826e04e3a4b4ee 2023-04-05 09:54:44,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/c9b9e70c7d64444f90dcf145e8e504e6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6 2023-04-05 09:54:44,853 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6, entries=150, sequenceid=33, filesize=11.6 K 2023-04-05 09:54:44,854 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5996f4881ea7494da5501c76439d71e1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1 2023-04-05 09:54:44,858 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1, entries=150, sequenceid=33, filesize=11.6 K 2023-04-05 09:54:44,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/7f87be579f414c3b8d826e04e3a4b4ee as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee 2023-04-05 09:54:44,864 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee, entries=150, sequenceid=33, filesize=11.6 K 2023-04-05 09:54:44,865 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8eb0aa9287d767302230619672527872 in 916ms, sequenceid=33, compaction requested=false 2023-04-05 09:54:44,865 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:44,865 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ec85c7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:44,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:44,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:44,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:44,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:44,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:44,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:44,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:44,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:44,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:44,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:44,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,880 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:44,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:44,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:44,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:44,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:44,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:44,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:44,883 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:44,883 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:44,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:44,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:44,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688544910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688544911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688544912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688544915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:44,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688544916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,983 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:44,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:44,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:44,987 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:44,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:44,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:44,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:44,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:44,990 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:44,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:44,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:44,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:44,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:44,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:44,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:44,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:44,991 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:44,991 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:44,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:44,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:44,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,993 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266dcb38[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:44,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:44,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:44,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:44,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:44,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:44,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:44,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:44,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:44,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,001 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,001 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,001 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,003 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,003 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,004 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,011 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,011 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688545016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688545017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688545017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688545019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688545020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,111 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,115 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,116 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,117 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,117 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,118 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,118 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,118 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:45,118 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:45,120 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ef8f25[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:45,122 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:45,122 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:45,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:45,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,128 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,128 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,128 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,128 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,128 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,129 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,129 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,130 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,138 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,138 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688545220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688545220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688545221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688545222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688545223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,238 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,242 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,246 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,246 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:45,246 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:45,247 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:45,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:45,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e06dc6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:45,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:45,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:45,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,258 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:45,258 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,258 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,262 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,262 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,264 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,264 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,264 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,269 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,269 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1dfe290a86364875af4401507673d8a9 2023-04-05 09:54:45,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/d46fc3f848d643dabdfd6091a9464629 2023-04-05 09:54:45,369 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,373 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,375 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,375 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,378 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,378 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,378 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:45,378 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,379 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:45,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766704c0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:45,381 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,382 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:45,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:45,384 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,387 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,387 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,387 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,387 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,387 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,389 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,390 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,394 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,394 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,394 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,494 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,500 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,514 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,514 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,514 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:45,514 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9b253f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:45,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:45,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,521 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:45,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:45,524 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:45,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688545523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688545524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688545525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,528 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,529 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,529 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,529 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,529 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688545526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,531 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,531 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,531 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,532 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,534 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688545533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,538 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,538 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,538 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,538 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:45,538 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,638 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,654 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,660 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,660 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,680 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,680 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,680 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:45,680 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:45,682 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:45,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7043136e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:45,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:45,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:45,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:45,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:45,696 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:45,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:45,701 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:45,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:45,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:45,705 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:45,706 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:45,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:45,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:45,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/b5d65ebcad9741398078922d02f400e9 2023-04-05 09:54:45,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1dfe290a86364875af4401507673d8a9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9 2023-04-05 09:54:45,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 09:54:45,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/d46fc3f848d643dabdfd6091a9464629 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629 2023-04-05 09:54:45,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 09:54:45,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/b5d65ebcad9741398078922d02f400e9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9 2023-04-05 09:54:45,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 09:54:45,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8eb0aa9287d767302230619672527872 in 881ms, sequenceid=50, compaction requested=true 2023-04-05 09:54:45,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:45,782 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:45,782 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:45,783 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:45,783 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:45,783 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:45,783 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:45,783 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,783 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,783 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=34.8 K 2023-04-05 09:54:45,783 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=34.8 K 2023-04-05 09:54:45,783 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting dc6beea24f7848e5a29f565f15bee1f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688482624 2023-04-05 09:54:45,784 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f28dfcccf16446449eb29c76bc5e1a34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688482624 2023-04-05 09:54:45,784 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c9b9e70c7d64444f90dcf145e8e504e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680688482660 2023-04-05 09:54:45,784 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5996f4881ea7494da5501c76439d71e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680688482660 2023-04-05 09:54:45,784 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1dfe290a86364875af4401507673d8a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:45,784 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d46fc3f848d643dabdfd6091a9464629, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:45,796 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#207 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:45,806 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:45,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:45,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:45,809 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:45,824 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:45,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:45,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:45,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:45,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:45,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:45,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:45,856 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:45,856 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:45,856 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-05 09:54:45,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:45,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:45,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:45,857 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:45,858 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:45,858 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:45,879 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/e28ace1e4c78475b8e7c1dcc1eb613d5 2023-04-05 09:54:45,882 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/8a4d3c35bc2547ecbb11ca59759ffbfe as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/8a4d3c35bc2547ecbb11ca59759ffbfe 2023-04-05 09:54:45,888 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 8a4d3c35bc2547ecbb11ca59759ffbfe(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:45,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:45,888 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=13, startTime=1680688485782; duration=0sec 2023-04-05 09:54:45,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:45,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:45,890 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:45,890 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:45,890 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:45,890 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=34.8 K 2023-04-05 09:54:45,890 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 59e4679671ff4876ab6d36f77d116f50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688482624 2023-04-05 09:54:45,892 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7f87be579f414c3b8d826e04e3a4b4ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680688482660 2023-04-05 09:54:45,892 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b5d65ebcad9741398078922d02f400e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:45,913 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:45,934 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/60bdf947ab50435ba250e6a6eef5bd0b 2023-04-05 09:54:45,962 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/ec73b2f970f64fb1bab658441334ea0a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ec73b2f970f64fb1bab658441334ea0a 2023-04-05 09:54:45,969 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into ec73b2f970f64fb1bab658441334ea0a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:45,969 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:45,969 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=13, startTime=1680688485782; duration=0sec 2023-04-05 09:54:45,969 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:45,991 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8de6e42d12334a9a84b7c1a6bb6c9359 2023-04-05 09:54:45,996 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/e28ace1e4c78475b8e7c1dcc1eb613d5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5 2023-04-05 09:54:46,002 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5, entries=150, sequenceid=66, filesize=11.6 K 2023-04-05 09:54:46,003 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/60bdf947ab50435ba250e6a6eef5bd0b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b 2023-04-05 09:54:46,009 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b, entries=150, sequenceid=66, filesize=11.6 K 2023-04-05 09:54:46,010 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8de6e42d12334a9a84b7c1a6bb6c9359 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359 2023-04-05 09:54:46,014 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359, entries=150, sequenceid=66, filesize=11.6 K 2023-04-05 09:54:46,015 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 8eb0aa9287d767302230619672527872 in 159ms, sequenceid=66, compaction requested=false 2023-04-05 09:54:46,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:46,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407dcefd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,025 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,025 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,025 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,025 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,026 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,028 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,033 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,033 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:46,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:46,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:46,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:46,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:46,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:46,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:46,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:46,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688546048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688546048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688546049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688546050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688546051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,134 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,138 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,140 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef29336[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,147 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,147 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,149 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688546152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688546153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688546153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,153 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,154 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,154 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688546154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688546155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,159 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,159 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,159 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,159 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,159 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,163 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:46,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,167 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,167 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,250 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/c405fbbcc1184fa19eb157095a5e7835 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/c405fbbcc1184fa19eb157095a5e7835 2023-04-05 09:54:46,254 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into c405fbbcc1184fa19eb157095a5e7835(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:46,254 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:46,254 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=13, startTime=1680688485782; duration=0sec 2023-04-05 09:54:46,254 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:46,267 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,270 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,272 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,272 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,273 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,274 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,274 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,275 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,275 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,275 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,275 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,280 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7fd0a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,283 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,286 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,286 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,287 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,294 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,294 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,294 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,295 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688546354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688546354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688546355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688546357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688546357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,395 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,398 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,399 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,399 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,401 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,403 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,403 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1175b33[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,407 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,407 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,410 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,410 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,410 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,410 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,410 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,412 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,412 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,412 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,427 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,427 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1855a1bac02148e58ace9b1c0d0ec257 2023-04-05 09:54:46,527 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,530 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,530 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,530 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,530 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,531 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,531 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,533 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,534 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,534 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,534 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,534 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,536 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3367ca[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,541 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,544 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,544 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,551 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:46,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,551 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,551 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,651 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,654 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,656 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,656 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,658 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,658 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688546657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,659 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,659 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688546657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688546658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,660 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688546660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c23a39[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688546662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,671 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,672 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,679 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,679 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,779 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,782 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635bde26[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,790 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,790 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,792 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,794 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,794 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,794 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,794 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,794 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,796 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,796 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,796 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:46,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,801 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,801 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,802 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,801 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/eff232cacb8b4987a7fe89201c4813ff 2023-04-05 09:54:46,902 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:46,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:46,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:46,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:46,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:46,909 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:46,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:46,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:46,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:46,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:46,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:46,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:46,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,916 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58aecfbd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:46,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:46,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,919 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:46,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:46,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:46,924 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:46,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:46,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:46,924 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:46,925 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:46,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,930 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:46,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:46,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:46,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:46,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:46,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:46,931 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,931 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:46,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:46,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,031 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,035 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,035 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,035 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,035 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,035 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,035 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,036 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:47,039 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c97ee55[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,043 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,043 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,044 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,046 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,048 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,048 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,053 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,059 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,059 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,058 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,159 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688547160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688547160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,163 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,163 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,163 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688547162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,165 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688547166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:47,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688547166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,176 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,176 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,176 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:47,177 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,179 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a0120c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,186 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,189 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,189 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,189 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,189 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,189 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,191 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,191 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,191 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,194 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,197 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,197 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:47,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e 2023-04-05 09:54:47,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1855a1bac02148e58ace9b1c0d0ec257 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257 2023-04-05 09:54:47,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257, entries=200, sequenceid=80, filesize=13.9 K 2023-04-05 09:54:47,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/eff232cacb8b4987a7fe89201c4813ff as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff 2023-04-05 09:54:47,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff, entries=150, sequenceid=80, filesize=11.6 K 2023-04-05 09:54:47,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e 2023-04-05 09:54:47,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e, entries=150, sequenceid=80, filesize=11.6 K 2023-04-05 09:54:47,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8eb0aa9287d767302230619672527872 in 1254ms, sequenceid=80, compaction requested=true 2023-04-05 09:54:47,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:47,289 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:47,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:47,289 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:47,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:47,290 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:47,291 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:47,291 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,291 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/8a4d3c35bc2547ecbb11ca59759ffbfe, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=37.2 K 2023-04-05 09:54:47,291 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:47,291 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:47,292 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,292 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/c405fbbcc1184fa19eb157095a5e7835, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=34.9 K 2023-04-05 09:54:47,292 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8a4d3c35bc2547ecbb11ca59759ffbfe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:47,292 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c405fbbcc1184fa19eb157095a5e7835, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:47,292 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e28ace1e4c78475b8e7c1dcc1eb613d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680688484909 2023-04-05 09:54:47,293 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 60bdf947ab50435ba250e6a6eef5bd0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680688484909 2023-04-05 09:54:47,293 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1855a1bac02148e58ace9b1c0d0ec257, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:47,293 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting eff232cacb8b4987a7fe89201c4813ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:47,297 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,301 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,313 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,324 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:47,325 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:47,325 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:47,354 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/21e3fc71dfb04ad9921e490821abb6dd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/21e3fc71dfb04ad9921e490821abb6dd 2023-04-05 09:54:47,359 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into 21e3fc71dfb04ad9921e490821abb6dd(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:47,359 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,359 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=13, startTime=1680688487289; duration=0sec 2023-04-05 09:54:47,359 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:47,359 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:47,360 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:47,360 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:47,360 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,360 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ec73b2f970f64fb1bab658441334ea0a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=34.9 K 2023-04-05 09:54:47,361 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ec73b2f970f64fb1bab658441334ea0a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680688484792 2023-04-05 09:54:47,361 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8de6e42d12334a9a84b7c1a6bb6c9359, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680688484909 2023-04-05 09:54:47,362 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f0c2bf7aed8d4717b2e32f6e2fe2c52e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:47,364 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1ba0596d93d9455ca4719df0951a2b8b 2023-04-05 09:54:47,365 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/901105d6bfc34dfab1f4f449aad10cd4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/901105d6bfc34dfab1f4f449aad10cd4 2023-04-05 09:54:47,369 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:47,373 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 901105d6bfc34dfab1f4f449aad10cd4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:47,373 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,373 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=13, startTime=1680688487289; duration=0sec 2023-04-05 09:54:47,373 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:47,395 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/5d8cbc5ef39949968c5c45c9d22cebd1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/5d8cbc5ef39949968c5c45c9d22cebd1 2023-04-05 09:54:47,406 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/db8d424c897c45248177124adddb0aed 2023-04-05 09:54:47,406 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into 5d8cbc5ef39949968c5c45c9d22cebd1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:47,406 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,406 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=13, startTime=1680688487289; duration=0sec 2023-04-05 09:54:47,406 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:47,418 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8650e6fa50114466af845bd473a01798 2023-04-05 09:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1ba0596d93d9455ca4719df0951a2b8b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b 2023-04-05 09:54:47,431 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b, entries=150, sequenceid=101, filesize=11.6 K 2023-04-05 09:54:47,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/db8d424c897c45248177124adddb0aed as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed 2023-04-05 09:54:47,437 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed, entries=150, sequenceid=101, filesize=11.6 K 2023-04-05 09:54:47,438 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8650e6fa50114466af845bd473a01798 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798 2023-04-05 09:54:47,443 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798, entries=150, sequenceid=101, filesize=11.6 K 2023-04-05 09:54:47,444 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 8eb0aa9287d767302230619672527872 in 120ms, sequenceid=101, compaction requested=false 2023-04-05 09:54:47,444 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,444 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,445 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75374fd0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,450 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,451 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,455 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,455 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,458 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,458 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,458 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:47,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,464 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,464 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,564 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,568 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,569 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,572 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,572 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,572 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,572 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4aa5ec[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,579 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,579 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,582 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,582 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,584 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,584 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,584 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,592 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,593 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,693 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,697 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,699 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,701 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,701 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,701 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,701 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8fc0d96[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,707 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,712 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,715 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,715 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,717 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,717 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,717 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,718 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:47,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,722 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,722 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,822 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,841 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,842 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,843 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,844 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,846 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40044ef7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,852 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,852 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,853 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,853 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,853 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,856 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,860 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,860 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,866 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,873 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,873 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,874 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,874 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,974 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:47,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:47,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:47,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:47,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:47,977 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:47,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:47,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:47,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:47,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:47,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:47,981 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:47,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f02d0e8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:47,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:47,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,986 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:47,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:47,989 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:47,992 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:47,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:47,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:47,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:47,992 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:47,994 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,994 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,994 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:47,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:47,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:47,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:47,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:47,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:47,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:47,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:47,999 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:47,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:47,999 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:47,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,099 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,102 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1696121[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,116 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,118 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,119 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,121 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,122 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,123 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,125 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,126 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,126 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,126 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,126 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:48,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:48,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:48,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688548187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688488186, max timestamp=1680688488190Num uniques 859; 2023-04-05 09:54:48,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:54:48,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688488186, max timestamp=1680688488190Num uniques 859; 2023-04-05 09:54:48,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:54:48,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688488186, max timestamp=1680688488190Num uniques 859; 2023-04-05 09:54:48,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:54:48,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688548203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688548204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688548205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688548205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,226 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,232 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,234 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69115c27[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,242 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,243 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,246 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,246 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,246 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,246 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,246 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,248 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,249 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,253 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,253 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,253 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,254 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688548292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688548307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688548310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688548310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688548310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,354 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,369 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,369 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,390 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,392 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,392 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,395 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b80d404[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,399 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,399 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,400 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,400 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,401 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,402 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,405 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,405 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,410 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,414 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,414 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,414 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,414 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688548497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688548510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,514 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688548514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688548514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688548515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,517 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,526 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,526 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,527 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,527 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,528 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129cd00c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,528 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,529 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,529 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,529 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,530 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,533 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,533 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,534 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,534 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,534 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,535 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,536 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,539 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,539 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,540 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,540 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,540 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,540 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,540 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,540 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/a53c50ccc122498881441755a0fca4f5 2023-04-05 09:54:48,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/982ab2d6e1a346e986403da79bf46250 2023-04-05 09:54:48,640 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,643 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,644 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,644 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,647 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,647 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,647 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,647 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,649 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5836db17[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,650 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,651 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,652 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,652 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,657 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,659 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,659 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,659 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,666 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,666 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,766 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,770 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,771 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,771 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,772 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,772 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:48,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,773 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,773 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,773 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,773 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b255452[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,779 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,779 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,779 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,780 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,782 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,782 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,784 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,789 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,789 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,789 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688548801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688548814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688548817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688548818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:48,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688548818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,899 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:48,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:48,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:48,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:48,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:48,903 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:48,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:48,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:48,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,907 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,907 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,908 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:48,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:48,910 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,910 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:48,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:48,911 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:48,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:48,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:48,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:48,911 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:48,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@368887c6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:48,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:48,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,916 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:48,916 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,916 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:48,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:48,927 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:48,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:48,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:48,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:48,927 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:48,929 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,929 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,929 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:48,929 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:48,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:48,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:48,933 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:48,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:48,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:48,933 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:48,933 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8631334a4cc343d0bb64a7938dc7542e 2023-04-05 09:54:49,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/a53c50ccc122498881441755a0fca4f5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5 2023-04-05 09:54:49,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5, entries=200, sequenceid=121, filesize=13.9 K 2023-04-05 09:54:49,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/982ab2d6e1a346e986403da79bf46250 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250 2023-04-05 09:54:49,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250, entries=150, sequenceid=121, filesize=11.6 K 2023-04-05 09:54:49,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8631334a4cc343d0bb64a7938dc7542e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e 2023-04-05 09:54:49,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e, entries=150, sequenceid=121, filesize=11.6 K 2023-04-05 09:54:49,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 8eb0aa9287d767302230619672527872 in 847ms, sequenceid=121, compaction requested=true 2023-04-05 09:54:49,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:49,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:49,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:49,019 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:49,019 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:49,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:49,020 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:49,020 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:49,020 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:49,020 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:49,021 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,021 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,021 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/901105d6bfc34dfab1f4f449aad10cd4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=37.3 K 2023-04-05 09:54:49,021 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/21e3fc71dfb04ad9921e490821abb6dd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.0 K 2023-04-05 09:54:49,031 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 901105d6bfc34dfab1f4f449aad10cd4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:49,032 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 21e3fc71dfb04ad9921e490821abb6dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:49,032 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1ba0596d93d9455ca4719df0951a2b8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688486046 2023-04-05 09:54:49,032 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting db8d424c897c45248177124adddb0aed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688486046 2023-04-05 09:54:49,032 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a53c50ccc122498881441755a0fca4f5, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:49,033 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 982ab2d6e1a346e986403da79bf46250, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:49,034 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:49,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:49,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:49,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:49,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:49,050 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:49,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:49,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:49,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,051 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:49,051 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:49,051 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,051 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:49,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:49,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:49,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,054 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:49,056 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,056 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:49,056 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 09:54:49,056 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:49,070 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/3e9551e60d2c42a294715e4f389f7e3d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/3e9551e60d2c42a294715e4f389f7e3d 2023-04-05 09:54:49,074 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 3e9551e60d2c42a294715e4f389f7e3d(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:49,074 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:49,074 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=13, startTime=1680688489019; duration=0sec 2023-04-05 09:54:49,075 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:49,075 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:49,076 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:49,076 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:49,076 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,076 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/5d8cbc5ef39949968c5c45c9d22cebd1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.0 K 2023-04-05 09:54:49,076 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5d8cbc5ef39949968c5c45c9d22cebd1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680688486032 2023-04-05 09:54:49,077 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8650e6fa50114466af845bd473a01798, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688486046 2023-04-05 09:54:49,077 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/36dd1ef54660475db0a168654d87eeac 2023-04-05 09:54:49,077 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8631334a4cc343d0bb64a7938dc7542e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:49,105 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:49,130 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/215e1446c9e54625a1b781e60003d995 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/215e1446c9e54625a1b781e60003d995 2023-04-05 09:54:49,135 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into 215e1446c9e54625a1b781e60003d995(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:49,135 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:49,135 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=13, startTime=1680688489019; duration=0sec 2023-04-05 09:54:49,136 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:49,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688549323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688549323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688549324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688549325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688549325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688549427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688549427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688549428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688549428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,467 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/ca64b1769cce4319b758ddba53850599 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ca64b1769cce4319b758ddba53850599 2023-04-05 09:54:49,471 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into ca64b1769cce4319b758ddba53850599(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:49,471 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:49,471 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=13, startTime=1680688489019; duration=0sec 2023-04-05 09:54:49,471 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:49,512 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/7b5983fc260a49758280bc9b16c87e26 2023-04-05 09:54:49,522 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/85664e7de51841c7bde69230cd797507 2023-04-05 09:54:49,529 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/36dd1ef54660475db0a168654d87eeac as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac 2023-04-05 09:54:49,534 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac, entries=150, sequenceid=155, filesize=11.8 K 2023-04-05 09:54:49,535 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/7b5983fc260a49758280bc9b16c87e26 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26 2023-04-05 09:54:49,545 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26, entries=150, sequenceid=155, filesize=11.8 K 2023-04-05 09:54:49,547 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/85664e7de51841c7bde69230cd797507 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507 2023-04-05 09:54:49,554 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507, entries=150, sequenceid=155, filesize=11.8 K 2023-04-05 09:54:49,554 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 8eb0aa9287d767302230619672527872 in 498ms, sequenceid=155, compaction requested=false 2023-04-05 09:54:49,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:49,555 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:49,557 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,557 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42603dab[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:49,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:49,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:49,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:49,565 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:49,565 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:49,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:49,577 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:49,577 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:49,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:49,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:49,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:49,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688549670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688549671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688549672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688549672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,677 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:49,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:49,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:49,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:49,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:49,688 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:49,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:49,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:49,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,690 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,691 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,691 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:49,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:49,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:49,703 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,703 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:49,703 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:49,703 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:49,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,709 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,709 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce0b893[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:49,710 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:49,710 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:49,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:49,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:49,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:49,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:49,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:49,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:49,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:49,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:49,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,724 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,725 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:49,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:49,725 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:49,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688549775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688549775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688549776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688549777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,825 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:49,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:49,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:49,829 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:49,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:49,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:49,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:49,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:49,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:49,836 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,836 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:49,836 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:49,837 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:49,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d76fe99[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:49,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:49,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:49,843 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:49,846 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:49,850 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:49,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:49,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:49,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:49,850 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:49,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,866 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:49,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,867 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:49,867 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:49,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:49,967 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:49,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:49,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:49,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:49,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:49,970 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:49,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:49,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:49,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:49,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:49,975 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,975 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:49,975 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:49,975 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,977 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f32f846[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:49,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:49,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,980 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:49,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688549979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:49,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:49,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688549980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688549980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:49,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688549981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:49,984 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:49,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:49,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:49,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:49,984 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:49,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:49,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:49,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:49,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:49,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:49,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,992 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:49,992 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:49,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:49,991 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:49,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:49,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:49,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:49,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:50,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/9307dbcf70e447709bfaded7cd042c1a 2023-04-05 09:54:50,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/33e14f0dfe8e46e1b8ae8efeb7e3fece 2023-04-05 09:54:50,092 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:50,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:50,095 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:50,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a90b1112076a47b6aaac3e9a05630d54 2023-04-05 09:54:50,095 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:50,095 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:50,096 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:50,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:50,106 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:50,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:50,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/9307dbcf70e447709bfaded7cd042c1a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a 2023-04-05 09:54:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:50,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a, entries=150, sequenceid=173, filesize=11.8 K 2023-04-05 09:54:50,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/33e14f0dfe8e46e1b8ae8efeb7e3fece as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece 2023-04-05 09:54:50,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:50,121 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,121 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:50,121 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:50,121 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece, entries=150, sequenceid=173, filesize=11.8 K 2023-04-05 09:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:50,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a90b1112076a47b6aaac3e9a05630d54 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54 2023-04-05 09:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cfc096e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:50,125 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:50,125 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54, entries=150, sequenceid=173, filesize=11.8 K 2023-04-05 09:54:50,129 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 8eb0aa9287d767302230619672527872 in 492ms, sequenceid=173, compaction requested=true 2023-04-05 09:54:50,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:50,129 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:50,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:50,130 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:50,130 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:50,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:50,130 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:50,130 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:50,130 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:50,131 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:50,131 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,131 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,131 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,131 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ca64b1769cce4319b758ddba53850599, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.4 K 2023-04-05 09:54:50,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,131 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/3e9551e60d2c42a294715e4f389f7e3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.4 K 2023-04-05 09:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:50,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:50,131 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ca64b1769cce4319b758ddba53850599, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:50,131 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 3e9551e60d2c42a294715e4f389f7e3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:50,131 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 7b5983fc260a49758280bc9b16c87e26, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680688488200 2023-04-05 09:54:50,132 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 36dd1ef54660475db0a168654d87eeac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680688488200 2023-04-05 09:54:50,132 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 33e14f0dfe8e46e1b8ae8efeb7e3fece, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:50,132 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9307dbcf70e447709bfaded7cd042c1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:50,133 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:50,137 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:50,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:50,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:50,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:50,137 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:50,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,139 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,139 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:50,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:50,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:50,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:50,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:50,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:50,143 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:50,144 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:50,157 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:50,157 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:50,176 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/0878b32962954924a730bf71a84d4097 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0878b32962954924a730bf71a84d4097 2023-04-05 09:54:50,181 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into 0878b32962954924a730bf71a84d4097(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:50,181 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:50,181 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=13, startTime=1680688490130; duration=0sec 2023-04-05 09:54:50,181 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:50,181 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:50,182 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:50,182 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:50,182 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,182 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/215e1446c9e54625a1b781e60003d995, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.4 K 2023-04-05 09:54:50,183 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 215e1446c9e54625a1b781e60003d995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680688488167 2023-04-05 09:54:50,183 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 85664e7de51841c7bde69230cd797507, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680688488200 2023-04-05 09:54:50,184 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a90b1112076a47b6aaac3e9a05630d54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:50,202 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:50,226 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/cf82b9bd1f5a46a1bde44cf49ff948e3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cf82b9bd1f5a46a1bde44cf49ff948e3 2023-04-05 09:54:50,232 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into cf82b9bd1f5a46a1bde44cf49ff948e3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:50,232 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:50,232 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=13, startTime=1680688490130; duration=0sec 2023-04-05 09:54:50,232 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:50,244 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:50,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:50,247 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:50,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:50,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:50,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:50,248 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,248 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:50,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:50,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:50,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:50,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:50,256 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:50,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:50,257 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:50,266 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/f51c5da9af9342cda1b3558dad5911be 2023-04-05 09:54:50,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:50,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688550291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688550292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688550292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688550293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688550336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688550395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688550395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688550395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688550395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,574 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/c1d09d3026c2436b934b59ae0b3de561 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c1d09d3026c2436b934b59ae0b3de561 2023-04-05 09:54:50,579 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into c1d09d3026c2436b934b59ae0b3de561(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:50,579 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:50,579 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=13, startTime=1680688490129; duration=0sec 2023-04-05 09:54:50,579 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:50,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688550599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688550599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688550599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688550600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,678 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/029114b0246d4e7c8ec2c4f2fa97bf6c 2023-04-05 09:54:50,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688550902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688550902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688550902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:50,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688550903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,089 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/4e366e4714ab4fc1bbb59004e47c371c 2023-04-05 09:54:51,093 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/f51c5da9af9342cda1b3558dad5911be as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be 2023-04-05 09:54:51,097 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 09:54:51,098 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/029114b0246d4e7c8ec2c4f2fa97bf6c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c 2023-04-05 09:54:51,102 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 09:54:51,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/4e366e4714ab4fc1bbb59004e47c371c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c 2023-04-05 09:54:51,106 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 09:54:51,107 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8eb0aa9287d767302230619672527872 in 851ms, sequenceid=193, compaction requested=false 2023-04-05 09:54:51,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:51,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:51,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f11c42[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:51,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:51,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:51,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,113 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:51,116 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:51,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:51,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:51,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:51,116 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:51,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:51,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,124 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:51,124 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:51,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:51,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,225 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:51,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:51,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:51,229 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:51,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:51,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:51,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,230 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:51,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:51,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:51,233 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,233 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:51,233 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 09:54:51,233 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,243 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/fd4193c2f8194d908512c62b31b5cf43 2023-04-05 09:54:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:51,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:51,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688551418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688551419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688551419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688551420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688551521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688551522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688551522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688551523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,662 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/ee327e23cdbc413482f5326dd0e70bb2 2023-04-05 09:54:51,672 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/79e2ce296cd3437aba9fd006d6559113 2023-04-05 09:54:51,676 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/fd4193c2f8194d908512c62b31b5cf43 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43 2023-04-05 09:54:51,680 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43, entries=150, sequenceid=208, filesize=11.8 K 2023-04-05 09:54:51,680 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/ee327e23cdbc413482f5326dd0e70bb2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2 2023-04-05 09:54:51,685 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2, entries=150, sequenceid=208, filesize=11.8 K 2023-04-05 09:54:51,685 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/79e2ce296cd3437aba9fd006d6559113 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113 2023-04-05 09:54:51,689 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113, entries=150, sequenceid=208, filesize=11.8 K 2023-04-05 09:54:51,690 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8eb0aa9287d767302230619672527872 in 457ms, sequenceid=208, compaction requested=true 2023-04-05 09:54:51,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:51,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:51,694 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff9bc53[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:51,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:51,696 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:51,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:51,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:51,701 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:51,701 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:51,701 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:51,701 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:51,701 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:51,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:51,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,708 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,708 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:51,708 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:51,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:51,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:51,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:51,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/6372b1b14eba41c487476de208f94b90 2023-04-05 09:54:51,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688551736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688551738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688551740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688551740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,808 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:51,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:51,811 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:51,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:51,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:51,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,812 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,813 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:51,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:51,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:51,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:51,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:51,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:51,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77518122[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:51,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:51,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:51,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,822 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:51,824 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:51,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:51,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:51,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:51,825 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:51,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,826 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:51,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,830 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,830 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:51,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688551840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688551844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688551844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:51,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688551845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,930 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:51,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:51,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:51,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:51,933 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:51,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:51,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:51,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:51,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:51,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:51,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:51,939 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b8a41df[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:51,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:51,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,942 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:51,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:51,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,943 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:51,946 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:51,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:51,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:51,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:51,946 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:51,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:51,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:51,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,948 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:51,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:51,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:51,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:51,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:51,952 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,952 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:51,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:51,953 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688552042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688552047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688552047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688552048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,053 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,055 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,055 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,055 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,061 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,061 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,061 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,061 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:52,063 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b45de0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:52,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:52,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,066 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:52,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:52,070 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:52,073 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:52,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:52,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:52,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:52,073 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:52,074 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,074 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,074 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,080 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:52,080 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/83ed4c77eba945a6b166085a7f9f343e 2023-04-05 09:54:52,180 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,183 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,186 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,186 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,186 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,186 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:52,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544d6441[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:52,190 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:52,190 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,191 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:52,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:52,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:52,195 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:52,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:52,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:52,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:52,195 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:52,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,197 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,198 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,201 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:52,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:52,201 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,202 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:52,202 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:52,302 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,305 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,307 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,309 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,309 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,309 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,309 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:52,311 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feeb77b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:52,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:52,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:52,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,317 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:52,319 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:52,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:52,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:52,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:52,319 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:52,321 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,321 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,321 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,321 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:52,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:52,325 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:52,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,325 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:52,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688552347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688552350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688552350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688552351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,353 DEBUG [Thread-1643] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:54:52,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688552351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,426 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,430 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,431 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,434 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,434 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,434 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,434 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:52,442 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66238fe0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:52,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:52,449 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:52,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:52,451 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:52,453 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:52,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:52,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:52,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:52,453 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:52,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:52,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,465 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:52,465 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/78463f7774284af299c8416ebbf80b54 2023-04-05 09:54:52,565 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/6372b1b14eba41c487476de208f94b90 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90 2023-04-05 09:54:52,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,569 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,571 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,571 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90, entries=250, sequenceid=228, filesize=16.4 K 2023-04-05 09:54:52,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/83ed4c77eba945a6b166085a7f9f343e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e 2023-04-05 09:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,577 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,577 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:52,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e, entries=150, sequenceid=228, filesize=11.8 K 2023-04-05 09:54:52,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/78463f7774284af299c8416ebbf80b54 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54 2023-04-05 09:54:52,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54, entries=150, sequenceid=228, filesize=11.8 K 2023-04-05 09:54:52,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 8eb0aa9287d767302230619672527872 in 859ms, sequenceid=228, compaction requested=true 2023-04-05 09:54:52,586 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:52,586 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:52,586 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:52,586 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:52,586 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:52,586 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:52,587 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:52,588 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53334 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:52,588 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:52,588 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:52,588 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,588 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,588 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0878b32962954924a730bf71a84d4097, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=47.4 K 2023-04-05 09:54:52,588 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c1d09d3026c2436b934b59ae0b3de561, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=52.1 K 2023-04-05 09:54:52,588 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0878b32962954924a730bf71a84d4097, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:52,589 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 029114b0246d4e7c8ec2c4f2fa97bf6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680688489665 2023-04-05 09:54:52,589 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c1d09d3026c2436b934b59ae0b3de561, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:52,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,589 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,589 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ee327e23cdbc413482f5326dd0e70bb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680688490290 2023-04-05 09:54:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:52,590 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f51c5da9af9342cda1b3558dad5911be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680688489665 2023-04-05 09:54:52,590 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 83ed4c77eba945a6b166085a7f9f343e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,590 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting fd4193c2f8194d908512c62b31b5cf43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680688490290 2023-04-05 09:54:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,590 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6372b1b14eba41c487476de208f94b90, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ec8599[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:52,591 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:52,591 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:52,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:52,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,600 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#246 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:52,601 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:52,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:52,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:52,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:52,601 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:52,602 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#247 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:52,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:52,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:52,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:52,610 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/cb2fc4a69a7d417296b69342e2fbc91b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/cb2fc4a69a7d417296b69342e2fbc91b 2023-04-05 09:54:52,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:52,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:52,614 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/92a7401fbe5a40b7a375cd62c736d0ff as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/92a7401fbe5a40b7a375cd62c736d0ff 2023-04-05 09:54:52,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:52,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:52,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:52,615 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:52,615 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:52,616 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into cb2fc4a69a7d417296b69342e2fbc91b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:52,616 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:52,616 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=12, startTime=1680688492586; duration=0sec 2023-04-05 09:54:52,616 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:52,616 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:52,618 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:52,618 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:52,618 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,618 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cf82b9bd1f5a46a1bde44cf49ff948e3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=47.4 K 2023-04-05 09:54:52,619 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cf82b9bd1f5a46a1bde44cf49ff948e3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680688489322 2023-04-05 09:54:52,619 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4e366e4714ab4fc1bbb59004e47c371c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680688489665 2023-04-05 09:54:52,619 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 79e2ce296cd3437aba9fd006d6559113, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680688490290 2023-04-05 09:54:52,619 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 78463f7774284af299c8416ebbf80b54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:52,620 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 92a7401fbe5a40b7a375cd62c736d0ff(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:52,620 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:52,620 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=12, startTime=1680688492586; duration=0sec 2023-04-05 09:54:52,620 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:52,627 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:52,715 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:52,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:52,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:52,719 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:52,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:52,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:52,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:52,720 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:52,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:52,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:52,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:52,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:52,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:52,723 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:52,723 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:52,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:52,855 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:52,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688552869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688552869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688552872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688552873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688552973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688552974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688552975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688552976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,038 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/cedb077e8d0a42f9accf92ad890bcc8a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cedb077e8d0a42f9accf92ad890bcc8a 2023-04-05 09:54:53,043 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into cedb077e8d0a42f9accf92ad890bcc8a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:53,043 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:53,043 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=12, startTime=1680688492586; duration=0sec 2023-04-05 09:54:53,043 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:53,131 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/f068f02061314e329aa22ae4c44ea7f4 2023-04-05 09:54:53,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688553176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1680688553176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688553177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688553179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688553479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688553480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1680688553482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688553482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,542 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e4a0730ee48a4742b04c4ccc5243b6f5 2023-04-05 09:54:53,953 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/82e818aceac24081be000941624f13d1 2023-04-05 09:54:53,957 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/f068f02061314e329aa22ae4c44ea7f4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4 2023-04-05 09:54:53,961 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4, entries=150, sequenceid=243, filesize=11.8 K 2023-04-05 09:54:53,962 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e4a0730ee48a4742b04c4ccc5243b6f5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5 2023-04-05 09:54:53,965 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5, entries=150, sequenceid=243, filesize=11.8 K 2023-04-05 09:54:53,966 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/82e818aceac24081be000941624f13d1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1 2023-04-05 09:54:53,970 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1, entries=150, sequenceid=243, filesize=11.8 K 2023-04-05 09:54:53,970 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8eb0aa9287d767302230619672527872 in 1248ms, sequenceid=243, compaction requested=false 2023-04-05 09:54:53,971 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:53,971 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:53,972 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:53,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:53,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3c98e9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:53,975 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:53,975 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,976 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:53,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:53,977 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:53,980 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:53,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:53,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:53,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:53,980 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:53,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:53,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,983 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:53,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:53,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:53,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:53,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:54:53,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:53,987 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:53,987 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:53,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:53,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:53,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:53,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688492867, max timestamp=1680688493986Num uniques 859; 2023-04-05 09:54:53,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:53,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688492867, max timestamp=1680688493986Num uniques 859; 2023-04-05 09:54:53,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:53,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/536ea165602b4ff6a0454842d29f767b 2023-04-05 09:54:54,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688554005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688554006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688493997, max timestamp=1680688494008Num uniques 859; 2023-04-05 09:54:54,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:54,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688493997, max timestamp=1680688494008Num uniques 859; 2023-04-05 09:54:54,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:54,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688493997, max timestamp=1680688494008Num uniques 859; 2023-04-05 09:54:54,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:54,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688554037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688554037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,088 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:54,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:54,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:54,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:54,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:54,091 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:54,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:54,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:54,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,092 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,093 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,093 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:54,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:54,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:54,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:54,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:54,094 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:54,096 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1733ab5a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:54,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:54,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:54,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55680 deadline: 1680688554110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:54,112 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:54,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:54,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:54,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:54,112 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:54,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688554110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:54,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,119 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,119 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:54,119 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688554142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688554142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,219 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:54,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:54,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:54,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:54,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:54,223 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:54,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:54,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:54,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,224 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:54,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:54,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:54,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:54,227 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,227 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:54,227 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:54,227 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:54,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:54,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:54,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:54,229 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,230 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,230 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:54,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d157ad3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:54,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:54,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:54,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:54,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:54,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:54,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:54,244 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:54,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:54,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:54,244 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:54,244 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:54,246 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,246 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,246 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,246 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,249 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,249 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:54,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,250 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:54,250 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:54,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688554312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55708 deadline: 1680688554314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688554344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688554344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,350 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:54,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:54,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:54,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:54,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:54,354 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:54,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:54,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:54,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,358 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,358 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:54,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:54,360 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,360 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:54,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b43faa5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:54,366 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:54,366 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,367 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:54,367 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,368 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:54,369 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:54,372 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:54,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:54,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:54,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:54,372 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:54,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,376 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,376 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:54,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:54,392 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,392 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,393 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:54,393 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:54,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:54,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/81aacafe59bd46ccaf2fd762e7e5589c 2023-04-05 09:54:54,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a1c019995b3e4de4b4d37caf37d50102 2023-04-05 09:54:54,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/536ea165602b4ff6a0454842d29f767b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b 2023-04-05 09:54:54,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b, entries=150, sequenceid=265, filesize=11.9 K 2023-04-05 09:54:54,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/81aacafe59bd46ccaf2fd762e7e5589c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c 2023-04-05 09:54:54,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c, entries=150, sequenceid=265, filesize=11.9 K 2023-04-05 09:54:54,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a1c019995b3e4de4b4d37caf37d50102 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102 2023-04-05 09:54:54,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102, entries=150, sequenceid=265, filesize=11.9 K 2023-04-05 09:54:54,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=127.47 KB/130530 for 8eb0aa9287d767302230619672527872 in 453ms, sequenceid=265, compaction requested=true 2023-04-05 09:54:54,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:54,440 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:54,440 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:54,440 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:54,440 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:54,440 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:54,441 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:54,441 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:54,441 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:54,441 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,442 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:54,442 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/92a7401fbe5a40b7a375cd62c736d0ff, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.9 K 2023-04-05 09:54:54,442 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,442 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/cb2fc4a69a7d417296b69342e2fbc91b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.9 K 2023-04-05 09:54:54,442 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 92a7401fbe5a40b7a375cd62c736d0ff, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:54,442 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cb2fc4a69a7d417296b69342e2fbc91b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:54,443 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f068f02061314e329aa22ae4c44ea7f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688491736 2023-04-05 09:54:54,443 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e4a0730ee48a4742b04c4ccc5243b6f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688491736 2023-04-05 09:54:54,443 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 536ea165602b4ff6a0454842d29f767b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:54,444 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 81aacafe59bd46ccaf2fd762e7e5589c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:54,451 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:54,456 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:54,468 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/4ae04402e7f7449094cf3d53550c90d0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4ae04402e7f7449094cf3d53550c90d0 2023-04-05 09:54:54,474 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 4ae04402e7f7449094cf3d53550c90d0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:54,474 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:54,474 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=13, startTime=1680688494440; duration=0sec 2023-04-05 09:54:54,474 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:54,474 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:54:54,475 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:54:54,475 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:54,475 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,475 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cedb077e8d0a42f9accf92ad890bcc8a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=35.9 K 2023-04-05 09:54:54,476 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting cedb077e8d0a42f9accf92ad890bcc8a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680688491414 2023-04-05 09:54:54,476 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 82e818aceac24081be000941624f13d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688491736 2023-04-05 09:54:54,476 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a1c019995b3e4de4b4d37caf37d50102, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:54,482 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:54,493 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:54,494 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8f4b16ccb9214831896bdb558021b338 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8f4b16ccb9214831896bdb558021b338 2023-04-05 09:54:54,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:54,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:54,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:54,496 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:54,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:54,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:54,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,498 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:54,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:54,500 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into 8f4b16ccb9214831896bdb558021b338(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:54,500 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:54,500 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=13, startTime=1680688494440; duration=0sec 2023-04-05 09:54:54,501 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:54,504 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:54,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:54,505 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:54,518 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/d4567239cbd141e08a3e4150ce500a3d 2023-04-05 09:54:54,539 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/02d2a755f04746818d12d6a2b39654fd 2023-04-05 09:54:54,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:54,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:54,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688554628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688554632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688554646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688554647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688554733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688554736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,870 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e7ee140cd78f42639e4f28ea38adeb15 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e7ee140cd78f42639e4f28ea38adeb15 2023-04-05 09:54:54,875 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into e7ee140cd78f42639e4f28ea38adeb15(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:54,875 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:54,875 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=13, startTime=1680688494440; duration=0sec 2023-04-05 09:54:54,876 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:54,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688554937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:54,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688554940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,951 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/fe87fffbc5404808ba9ba8720566d605 2023-04-05 09:54:54,956 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/d4567239cbd141e08a3e4150ce500a3d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d 2023-04-05 09:54:54,961 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d, entries=150, sequenceid=303, filesize=11.9 K 2023-04-05 09:54:54,962 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/02d2a755f04746818d12d6a2b39654fd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd 2023-04-05 09:54:54,972 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd, entries=150, sequenceid=303, filesize=11.9 K 2023-04-05 09:54:54,973 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/fe87fffbc5404808ba9ba8720566d605 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605 2023-04-05 09:54:54,978 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605, entries=150, sequenceid=303, filesize=11.9 K 2023-04-05 09:54:54,979 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 8eb0aa9287d767302230619672527872 in 475ms, sequenceid=303, compaction requested=false 2023-04-05 09:54:54,979 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:54,979 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:54,981 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5211cef4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:54,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:54,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,984 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:54,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,986 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:54,988 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:54,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:54,988 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:54,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:54,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,993 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:54,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:54,997 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:54,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:54,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:54,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:54,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:54,997 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:54,997 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:54,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:54,997 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:55,098 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:55,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:55,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:55,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:55,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:55,101 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:55,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:55,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:55,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:55,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:55,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:55,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:55,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:55,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:55,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:55,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:55,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:55,105 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:55,106 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:55,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688495168, max timestamp=1680688495176Num uniques 859; 2023-04-05 09:54:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688495168, max timestamp=1680688495176Num uniques 859; 2023-04-05 09:54:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688495168, max timestamp=1680688495176Num uniques 859; 2023-04-05 09:54:55,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:55,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688555197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688555200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688555239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688555243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688555301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688555302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688555504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688555506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,513 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/31ded033f973466fa96adb2e14769f54 2023-04-05 09:54:55,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688555743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688555749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688555807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688555812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:55,923 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5bfff9b1d7f14ec29930f97c6663aa3d 2023-04-05 09:54:56,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688556313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688556318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,333 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8dd89ab7b4124e998d3f86a626c37946 2023-04-05 09:54:56,337 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/31ded033f973466fa96adb2e14769f54 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54 2023-04-05 09:54:56,347 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:54:56,348 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5bfff9b1d7f14ec29930f97c6663aa3d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d 2023-04-05 09:54:56,352 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:54:56,352 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/8dd89ab7b4124e998d3f86a626c37946 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946 2023-04-05 09:54:56,356 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 09:54:56,357 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8eb0aa9287d767302230619672527872 in 1251ms, sequenceid=315, compaction requested=true 2023-04-05 09:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:56,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f2d697[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:56,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:56,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:56,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:56,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 09:54:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:56,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:56,367 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:56,367 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:56,367 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:56,367 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:56,368 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:56,369 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,369 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,369 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,370 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,372 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:56,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:56,374 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:56,375 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:56,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688556386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,475 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:56,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:56,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:56,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:56,481 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:56,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:56,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:56,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:56,486 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,486 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:56,486 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:56,486 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:56,487 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef7ecb8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:56,490 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:56,491 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,492 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:56,492 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,492 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:56,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688556490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:56,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:56,498 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:56,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:56,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:56,498 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:56,498 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:56,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,500 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,501 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,507 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:56,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:56,508 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:56,508 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:56,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:56,608 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:56,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:56,611 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:56,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:56,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:56,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:56,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:56,615 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,615 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:56,615 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:56,615 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,617 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a85ab11[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:56,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:56,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:56,629 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:56,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:56,629 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:56,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,631 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,633 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,641 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:56,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,642 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:56,642 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:56,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688556694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,742 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:56,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:56,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:56,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:56,745 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:56,745 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,745 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:56,745 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:56,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:56,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:56,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:56,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:56,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688556747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,750 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263fbcdb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:56,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:56,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:56,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:56,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688556755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:56,760 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:56,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:56,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:56,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:56,760 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:56,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,762 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,763 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,765 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:56,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,765 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,766 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:56,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:56,766 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:56,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/4837009ed2c3485087270c46c97b22cc 2023-04-05 09:54:56,866 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:56,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:56,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:56,869 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:56,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:56,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:56,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:56,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:56,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:56,873 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,873 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:56,873 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:56,873 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:56,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,876 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683790e5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:56,885 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:56,885 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,886 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:56,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,887 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:56,888 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:56,893 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:56,893 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:56,893 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:56,893 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:56,893 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:56,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:56,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,895 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,896 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,898 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:56,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:56,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:56,907 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:56,907 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:56,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:56,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688556999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,007 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:57,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:57,010 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:57,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:57,010 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:57,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,012 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,012 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:57,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:57,012 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:57,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:57,016 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49042377[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:57,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:57,018 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,019 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:57,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:57,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:57,023 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:57,024 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:57,024 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:57,024 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:57,024 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:57,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,028 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:57,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:57,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:57,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,032 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:57,032 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:57,132 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:57,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:57,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:57,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:57,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:57,135 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:57,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:57,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:57,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:57,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:57,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,139 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:57,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:57,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:57,140 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,140 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:57,140 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:57,140 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,142 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:57,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0ec465[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:57,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:57,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:57,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:57,148 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:57,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:57,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:57,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:57,149 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:57,151 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,151 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:57,151 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:57,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,152 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,153 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,155 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:57,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,156 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:57,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:57,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:57,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:57,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:57,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:57,156 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:57,156 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:57,156 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:57,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:57,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/91c5314d226046ffb4820e9017f045ef 2023-04-05 09:54:57,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a412177e378d4c5fb7adf864479fd98c 2023-04-05 09:54:57,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/4837009ed2c3485087270c46c97b22cc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc 2023-04-05 09:54:57,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc, entries=200, sequenceid=351, filesize=14.3 K 2023-04-05 09:54:57,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/91c5314d226046ffb4820e9017f045ef as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef 2023-04-05 09:54:57,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef, entries=150, sequenceid=351, filesize=11.9 K 2023-04-05 09:54:57,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/a412177e378d4c5fb7adf864479fd98c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c 2023-04-05 09:54:57,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c, entries=150, sequenceid=351, filesize=11.9 K 2023-04-05 09:54:57,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 8eb0aa9287d767302230619672527872 in 856ms, sequenceid=351, compaction requested=true 2023-04-05 09:54:57,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:57,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:57,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:57,220 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:57,220 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:57,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:57,221 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:57,222 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:57,222 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:54:57,222 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:54:57,222 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,222 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,222 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e7ee140cd78f42639e4f28ea38adeb15, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=48.2 K 2023-04-05 09:54:57,222 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4ae04402e7f7449094cf3d53550c90d0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=50.6 K 2023-04-05 09:54:57,222 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e7ee140cd78f42639e4f28ea38adeb15, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:57,222 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4ae04402e7f7449094cf3d53550c90d0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:57,223 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 02d2a755f04746818d12d6a2b39654fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680688494030 2023-04-05 09:54:57,223 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d4567239cbd141e08a3e4150ce500a3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680688494030 2023-04-05 09:54:57,223 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5bfff9b1d7f14ec29930f97c6663aa3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688494622 2023-04-05 09:54:57,223 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 31ded033f973466fa96adb2e14769f54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688494622 2023-04-05 09:54:57,224 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 91c5314d226046ffb4820e9017f045ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:54:57,224 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4837009ed2c3485087270c46c97b22cc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:54:57,234 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#267 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:57,241 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:57,248 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/037d5626719b43f8828b980390be7313 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/037d5626719b43f8828b980390be7313 2023-04-05 09:54:57,256 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:57,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:57,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:57,259 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:57,259 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,259 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:57,259 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:57,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,261 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,261 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:57,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:57,261 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,261 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:57,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:57,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:57,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:57,264 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:57,264 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:57,264 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,264 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/96e9bb246ab041188e0ec421bd343892 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/96e9bb246ab041188e0ec421bd343892 2023-04-05 09:54:57,264 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:57,264 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:57,265 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:57,265 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:57,265 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:57,265 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:57,265 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:57,266 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:57,269 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into 037d5626719b43f8828b980390be7313(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:57,269 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:57,269 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=12, startTime=1680688497220; duration=0sec 2023-04-05 09:54:57,269 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:54:57,269 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:54:57,271 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:54:57,271 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:54:57,271 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:57,271 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8f4b16ccb9214831896bdb558021b338, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=48.2 K 2023-04-05 09:54:57,272 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8f4b16ccb9214831896bdb558021b338, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680688492872 2023-04-05 09:54:57,272 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting fe87fffbc5404808ba9ba8720566d605, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680688494030 2023-04-05 09:54:57,272 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8dd89ab7b4124e998d3f86a626c37946, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680688494622 2023-04-05 09:54:57,273 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a412177e378d4c5fb7adf864479fd98c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:54:57,274 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 96e9bb246ab041188e0ec421bd343892(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:57,274 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:57,274 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=12, startTime=1680688497220; duration=0sec 2023-04-05 09:54:57,274 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:57,282 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:54:57,318 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/b3259f3a42cd42c7bbbf71c39821fe79 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b3259f3a42cd42c7bbbf71c39821fe79 2023-04-05 09:54:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:57,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:57,325 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into b3259f3a42cd42c7bbbf71c39821fe79(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:54:57,325 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:57,325 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=12, startTime=1680688497220; duration=0sec 2023-04-05 09:54:57,325 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688497337, max timestamp=1680688497343Num uniques 859; 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688497337, max timestamp=1680688497343Num uniques 859; 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688497337, max timestamp=1680688497343Num uniques 859; 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:57,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:54:57,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688557373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688557373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688557477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688557477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688557505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688557680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688557680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,703 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/5bfc682a068c4e0fb785de404a7c237f 2023-04-05 09:54:57,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688557984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:57,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:57,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688557984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,120 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5f3714d2848447a0a094954b760f8a83 2023-04-05 09:54:58,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688558489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688558489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688558509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,537 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/6c575db4ec2d4ee9890e7ba21e835e66 2023-04-05 09:54:58,541 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/5bfc682a068c4e0fb785de404a7c237f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f 2023-04-05 09:54:58,545 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f, entries=150, sequenceid=361, filesize=11.9 K 2023-04-05 09:54:58,545 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/5f3714d2848447a0a094954b760f8a83 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83 2023-04-05 09:54:58,549 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83, entries=150, sequenceid=361, filesize=11.9 K 2023-04-05 09:54:58,550 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/6c575db4ec2d4ee9890e7ba21e835e66 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66 2023-04-05 09:54:58,560 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66, entries=150, sequenceid=361, filesize=11.9 K 2023-04-05 09:54:58,560 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8eb0aa9287d767302230619672527872 in 1296ms, sequenceid=361, compaction requested=false 2023-04-05 09:54:58,560 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:58,561 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:58,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,563 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:58,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:58,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450c7a1b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:58,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:58,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:58,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:58,569 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:58,572 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:58,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:58,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:58,572 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:58,572 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:58,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:58,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:58,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:58,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:58,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,579 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:58,579 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:58,579 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:58,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:58,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:58,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:58,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:58,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:58,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,580 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:58,580 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:58,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:58,680 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:58,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:58,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:58,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:58,683 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:58,685 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:58,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:58,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:58,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:58,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:58,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:58,687 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:58,687 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:54:58,694 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/138b5780d974494cbd2e1ebc1c04353d 2023-04-05 09:54:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:58,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:58,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688558775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688558775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688558878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:58,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688558878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688559081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688559082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,117 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e16f351245e645da8cbcb6f67194f3da 2023-04-05 09:54:59,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688559384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688559384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688559491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688559494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,531 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/4c0f4e831b7b48369c9c05eb2fd89644 2023-04-05 09:54:59,535 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/138b5780d974494cbd2e1ebc1c04353d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d 2023-04-05 09:54:59,539 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d, entries=150, sequenceid=401, filesize=11.9 K 2023-04-05 09:54:59,539 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e16f351245e645da8cbcb6f67194f3da as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da 2023-04-05 09:54:59,542 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da, entries=150, sequenceid=401, filesize=11.9 K 2023-04-05 09:54:59,543 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/4c0f4e831b7b48369c9c05eb2fd89644 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644 2023-04-05 09:54:59,546 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644, entries=150, sequenceid=401, filesize=11.9 K 2023-04-05 09:54:59,547 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 8eb0aa9287d767302230619672527872 in 860ms, sequenceid=401, compaction requested=true 2023-04-05 09:54:59,547 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:54:59,547 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:54:59,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11686ae7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:54:59,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:54:59,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,552 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:54:59,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:54:59,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:54:59,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:59,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:54:59,557 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:54:59,557 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:54:59,557 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:54:59,557 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:54:59,557 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:54:59,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:54:59,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:54:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:54:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:54:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:54:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:59,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:59,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:59,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:54:59,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:54:59,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:54:59,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:59,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:54:59,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:54:59,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,564 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:54:59,564 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:54:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:59,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:54:59,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:54:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:54:59,664 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:54:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:54:59,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:54:59,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:54:59,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:54:59,667 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:54:59,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:59,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:54:59,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:54:59,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:59,668 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:54:59,668 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:54:59,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:59,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:54:59,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:54:59,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:54:59,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:54:59,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:54:59,670 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:54:59,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:54:59,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:54:59,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688499913, max timestamp=1680688499926Num uniques 859; 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688499913, max timestamp=1680688499926Num uniques 859; 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688499913, max timestamp=1680688499926Num uniques 859; 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:59,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:54:59,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688559958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:54:59,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:54:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688559958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688560062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688560062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,077 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/9c9cee466861475ba63ac42a51367eb3 2023-04-05 09:55:00,098 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/a0557980a5b64251b7362f11361d1d17 2023-04-05 09:55:00,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688560265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688560265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,517 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/216a5960343d40a899ac2e0b961d2f8d 2023-04-05 09:55:00,521 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/9c9cee466861475ba63ac42a51367eb3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3 2023-04-05 09:55:00,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55686 deadline: 1680688560521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,525 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3, entries=150, sequenceid=411, filesize=11.9 K 2023-04-05 09:55:00,526 DEBUG [Thread-1643] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:00,527 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/a0557980a5b64251b7362f11361d1d17 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17 2023-04-05 09:55:00,532 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17, entries=150, sequenceid=411, filesize=11.9 K 2023-04-05 09:55:00,533 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/216a5960343d40a899ac2e0b961d2f8d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d 2023-04-05 09:55:00,540 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d, entries=150, sequenceid=411, filesize=11.9 K 2023-04-05 09:55:00,541 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8eb0aa9287d767302230619672527872 in 871ms, sequenceid=411, compaction requested=true 2023-04-05 09:55:00,541 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:00,541 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:00,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:00,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799fec26[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:00,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:00,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:00,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:00,552 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:00,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:00,556 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:00,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:00,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:00,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:00,556 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:00,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,560 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:00,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:00,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:00,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,564 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:00,564 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:00,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688499952, max timestamp=1680688500571Num uniques 1018; 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688499952, max timestamp=1680688500571Num uniques 1018; 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688499952, max timestamp=1680688500571Num uniques 1018; 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-05 09:55:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:00,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:00,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:00,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/18c036564cd24c12bfd71a0a1a754694 2023-04-05 09:55:00,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688500594, max timestamp=1680688500608Num uniques 859; 2023-04-05 09:55:00,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688500594, max timestamp=1680688500608Num uniques 859; 2023-04-05 09:55:00,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:00,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:00,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688500594, max timestamp=1680688500608Num uniques 859; 2023-04-05 09:55:00,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:00,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/45799397adca437a8b7f84b2c3a2e885 2023-04-05 09:55:00,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688500642, max timestamp=1680688500654Num uniques 1018; 2023-04-05 09:55:00,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688500642, max timestamp=1680688500654Num uniques 1018; 2023-04-05 09:55:00,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688500642, max timestamp=1680688500654Num uniques 1018; 2023-04-05 09:55:00,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:00,664 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:00,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:00,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/b1594c8541ac4c94ba1f1105f25a319f 2023-04-05 09:55:00,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/18c036564cd24c12bfd71a0a1a754694 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694 2023-04-05 09:55:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:00,678 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:00,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:00,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:00,679 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,679 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694, entries=200, sequenceid=450, filesize=14.3 K 2023-04-05 09:55:00,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/45799397adca437a8b7f84b2c3a2e885 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885 2023-04-05 09:55:00,681 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:00,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:00,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:00,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:00,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:00,685 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:00,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885, entries=150, sequenceid=450, filesize=11.9 K 2023-04-05 09:55:00,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/b1594c8541ac4c94ba1f1105f25a319f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f 2023-04-05 09:55:00,686 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,687 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11da329c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:00,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:00,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,690 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:00,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:00,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:00,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:00,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:00,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:00,694 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:00,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,697 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f, entries=150, sequenceid=450, filesize=11.9 K 2023-04-05 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=114.05 KB/116790 for 8eb0aa9287d767302230619672527872 in 125ms, sequenceid=450, compaction requested=true 2023-04-05 09:55:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:00,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:00,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:00,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:00,699 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,700 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,704 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64154 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:00,704 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:55:00,704 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,704 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/96e9bb246ab041188e0ec421bd343892, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=62.7 K 2023-04-05 09:55:00,704 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:00,704 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:55:00,704 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,705 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/037d5626719b43f8828b980390be7313, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=60.3 K 2023-04-05 09:55:00,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,705 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 96e9bb246ab041188e0ec421bd343892, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:55:00,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,705 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:00,706 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:00,706 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 037d5626719b43f8828b980390be7313, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:55:00,706 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5bfc682a068c4e0fb785de404a7c237f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688496376 2023-04-05 09:55:00,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=120.76 KB heapSize=344.30 KB 2023-04-05 09:55:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:55:00,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:00,706 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5f3714d2848447a0a094954b760f8a83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688496376 2023-04-05 09:55:00,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:00,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:00,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:00,707 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e16f351245e645da8cbcb6f67194f3da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680688497368 2023-04-05 09:55:00,707 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 138b5780d974494cbd2e1ebc1c04353d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680688497368 2023-04-05 09:55:00,708 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a0557980a5b64251b7362f11361d1d17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680688498765 2023-04-05 09:55:00,708 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9c9cee466861475ba63ac42a51367eb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680688498765 2023-04-05 09:55:00,709 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 18c036564cd24c12bfd71a0a1a754694, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:00,709 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 45799397adca437a8b7f84b2c3a2e885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:00,726 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#283 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:00,729 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:00,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688560738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688560738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,806 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:00,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:00,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:00,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:00,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:00,809 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:00,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:00,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:00,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:00,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:00,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:00,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,812 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:00,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:00,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:00,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a05e1e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,818 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:00,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:00,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:00,822 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:00,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:00,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:00,822 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:00,822 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:00,824 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,824 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,824 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,824 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:00,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:00,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,828 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:00,828 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,828 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688560845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688560845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,928 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:00,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:00,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:00,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:00,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:00,936 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:00,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:00,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:00,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:00,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:00,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:00,940 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,940 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:00,940 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:00,940 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:00,942 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cd355f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:00,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:00,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,951 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,951 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:00,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:00,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:00,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:00,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:00,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:00,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:00,957 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:00,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:00,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:00,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:00,958 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:00,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:00,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:00,960 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,961 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:00,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:00,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:00,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:00,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:00,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:00,963 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:00,963 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:00,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:01,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688561050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688561051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,063 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:01,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:01,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:01,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:01,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:01,067 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:01,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:01,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:01,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:01,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,073 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:01,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:01,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:01,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:01,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:01,075 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:01,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:01,075 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:01,075 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:01,075 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:01,078 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63351b1c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:01,079 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:01,079 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,080 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,080 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:01,080 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:01,080 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:01,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:01,081 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:01,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:01,084 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:01,084 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:01,084 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:01,084 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:01,084 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:01,085 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,085 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:01,086 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:01,086 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:01,090 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:01,091 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:01,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:01,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:01,091 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:01,091 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:01,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:01,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/2fd675a4437346259857822deab8e7c4 2023-04-05 09:55:01,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/6c727a11b7474b9588d34b08af815871 2023-04-05 09:55:01,156 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/50c82b39f8c74f0885f80ccc7a6584e7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/50c82b39f8c74f0885f80ccc7a6584e7 2023-04-05 09:55:01,156 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/0162232909ea4fd497ad43ea04b2167d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0162232909ea4fd497ad43ea04b2167d 2023-04-05 09:55:01,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/6481a43311da4adc9b11ce6ec84e8b16 2023-04-05 09:55:01,161 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 50c82b39f8c74f0885f80ccc7a6584e7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:01,161 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:01,161 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=11, startTime=1680688500699; duration=0sec 2023-04-05 09:55:01,161 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:01,161 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:01,162 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into 0162232909ea4fd497ad43ea04b2167d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:01,162 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:01,162 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=11, startTime=1680688500699; duration=0sec 2023-04-05 09:55:01,162 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:01,163 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:01,163 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:55:01,164 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:01,164 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b3259f3a42cd42c7bbbf71c39821fe79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=60.3 K 2023-04-05 09:55:01,164 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b3259f3a42cd42c7bbbf71c39821fe79, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680688495186 2023-04-05 09:55:01,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/2fd675a4437346259857822deab8e7c4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4 2023-04-05 09:55:01,164 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6c575db4ec2d4ee9890e7ba21e835e66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688496376 2023-04-05 09:55:01,165 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4c0f4e831b7b48369c9c05eb2fd89644, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680688497368 2023-04-05 09:55:01,165 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 216a5960343d40a899ac2e0b961d2f8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680688498765 2023-04-05 09:55:01,165 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b1594c8541ac4c94ba1f1105f25a319f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:01,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4, entries=150, sequenceid=502, filesize=11.9 K 2023-04-05 09:55:01,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/6c727a11b7474b9588d34b08af815871 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871 2023-04-05 09:55:01,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871, entries=150, sequenceid=502, filesize=11.9 K 2023-04-05 09:55:01,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/6481a43311da4adc9b11ce6ec84e8b16 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16 2023-04-05 09:55:01,175 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#287 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:01,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16, entries=150, sequenceid=502, filesize=11.9 K 2023-04-05 09:55:01,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~344.25 KB/352512, currentSize=60.38 KB/61830 for 8eb0aa9287d767302230619672527872 in 471ms, sequenceid=502, compaction requested=false 2023-04-05 09:55:01,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:01,191 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:01,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:01,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:01,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:01,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:01,194 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:01,194 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:01,194 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:01,194 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:01,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,195 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:01,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:01,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:01,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:01,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:01,198 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:01,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:55:01,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:01,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688561387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688561389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688561491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688561492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1680688561509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,512 DEBUG [Thread-1635] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:01,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1680688561512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,515 DEBUG [Thread-1641] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:01,584 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/d2cdd3ea95154d25adfce48c9235c32d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d2cdd3ea95154d25adfce48c9235c32d 2023-04-05 09:55:01,589 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into d2cdd3ea95154d25adfce48c9235c32d(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:01,589 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:01,589 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=11, startTime=1680688500699; duration=0sec 2023-04-05 09:55:01,589 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:01,605 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1a4ba3e233ce4b6b94d0f1dbb1c20801 2023-04-05 09:55:01,615 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/68765d6674384d86b07dcf90be57f867 2023-04-05 09:55:01,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688561693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688561694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688561995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:01,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688561997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,025 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/d6b136d1e64a4c4e86e8682d5a1948ae 2023-04-05 09:55:02,030 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/1a4ba3e233ce4b6b94d0f1dbb1c20801 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801 2023-04-05 09:55:02,034 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801, entries=150, sequenceid=516, filesize=11.9 K 2023-04-05 09:55:02,034 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/68765d6674384d86b07dcf90be57f867 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867 2023-04-05 09:55:02,038 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867, entries=150, sequenceid=516, filesize=11.9 K 2023-04-05 09:55:02,038 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/d6b136d1e64a4c4e86e8682d5a1948ae as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae 2023-04-05 09:55:02,041 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae, entries=150, sequenceid=516, filesize=11.9 K 2023-04-05 09:55:02,042 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8eb0aa9287d767302230619672527872 in 844ms, sequenceid=516, compaction requested=true 2023-04-05 09:55:02,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:02,042 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:02,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@233b8b34[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:02,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:02,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,046 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:02,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:02,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:02,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:02,048 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:02,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,048 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:02,050 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:02,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:02,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:02,050 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:02,050 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:02,052 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,052 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,052 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:02,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:02,053 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:02,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,056 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,063 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:02,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:02,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:02,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:02,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:02,063 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:02,063 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:02,163 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:02,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:02,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:02,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:02,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:02,167 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:02,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:02,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:02,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:02,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:02,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:02,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:02,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:02,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. started... 2023-04-05 09:55:02,171 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:02,171 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,186 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/aa5e9cc7aac0401aa7a51e6e8995bf73 2023-04-05 09:55:02,196 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/0ca9f6e926d14282bd88092b1d57eaee 2023-04-05 09:55:02,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:55:02,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. as already flushing 2023-04-05 09:55:02,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:02,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55708 deadline: 1680688562542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55680 deadline: 1680688562542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,603 DEBUG [Thread-1652] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:62753 2023-04-05 09:55:02,603 DEBUG [Thread-1652] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,605 DEBUG [Thread-1648] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:62753 2023-04-05 09:55:02,605 DEBUG [Thread-1648] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,605 DEBUG [Thread-1646] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:62753 2023-04-05 09:55:02,606 DEBUG [Thread-1646] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,606 DEBUG [Thread-1650] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:62753 2023-04-05 09:55:02,606 DEBUG [Thread-1650] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,607 DEBUG [Thread-1654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:62753 2023-04-05 09:55:02,608 DEBUG [Thread-1654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,612 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/ac04a8a0237a4ade9f18d56a68f22097 2023-04-05 09:55:02,615 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/aa5e9cc7aac0401aa7a51e6e8995bf73 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73 2023-04-05 09:55:02,618 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73, entries=150, sequenceid=537, filesize=11.9 K 2023-04-05 09:55:02,618 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/0ca9f6e926d14282bd88092b1d57eaee as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee 2023-04-05 09:55:02,621 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee, entries=150, sequenceid=537, filesize=11.9 K 2023-04-05 09:55:02,622 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/ac04a8a0237a4ade9f18d56a68f22097 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097 2023-04-05 09:55:02,625 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097, entries=150, sequenceid=537, filesize=11.9 K 2023-04-05 09:55:02,625 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 8eb0aa9287d767302230619672527872 in 454ms, sequenceid=537, compaction requested=true 2023-04-05 09:55:02,625 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:02,625 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:02,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:02,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:02,627 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165e71e3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:02,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:02,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:02,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:02,639 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:02,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:02,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:02,641 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:02,641 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:02,641 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:02,641 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:02,641 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:02,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:02,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,642 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:02,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:02,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,643 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:02,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:02,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:02,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:02,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,645 DEBUG [Thread-1645] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:02,645 DEBUG [Thread-1645] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:02,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:02,645 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:02,646 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:02,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 8eb0aa9287d767302230619672527872 2023-04-05 09:55:02,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 09:55:02,647 DEBUG [Thread-1639] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:62753 2023-04-05 09:55:02,647 DEBUG [Thread-1639] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,647 DEBUG [Thread-1637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:62753 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:02,647 DEBUG [Thread-1637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:02,746 DEBUG [Thread-1645] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:02,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:03,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/ac7d2b8caa0240b89d3ea48a8387dd29 2023-04-05 09:55:03,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/db59b6f991cd490cbd2a461e359e1b3c 2023-04-05 09:55:03,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/bd173e5b02dc4212ab90bf899b22cb6c 2023-04-05 09:55:03,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/ac7d2b8caa0240b89d3ea48a8387dd29 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29 2023-04-05 09:55:03,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29, entries=150, sequenceid=552, filesize=11.9 K 2023-04-05 09:55:03,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/db59b6f991cd490cbd2a461e359e1b3c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c 2023-04-05 09:55:03,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c, entries=150, sequenceid=552, filesize=11.9 K 2023-04-05 09:55:03,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/bd173e5b02dc4212ab90bf899b22cb6c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c 2023-04-05 09:55:03,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c, entries=150, sequenceid=552, filesize=11.9 K 2023-04-05 09:55:03,885 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8eb0aa9287d767302230619672527872 in 1238ms, sequenceid=552, compaction requested=true 2023-04-05 09:55:03,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:03,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:03,886 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:03,886 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:03,886 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:03,886 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/A is initiating minor compaction (all files) 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/B is initiating minor compaction (all files) 2023-04-05 09:55:03,887 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/A in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:03,887 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/B in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:03,887 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/50c82b39f8c74f0885f80ccc7a6584e7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=60.4 K 2023-04-05 09:55:03,887 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0162232909ea4fd497ad43ea04b2167d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=60.4 K 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 50c82b39f8c74f0885f80ccc7a6584e7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:03,887 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0162232909ea4fd497ad43ea04b2167d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2fd675a4437346259857822deab8e7c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680688500695 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6c727a11b7474b9588d34b08af815871, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680688500695 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1a4ba3e233ce4b6b94d0f1dbb1c20801, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680688500729 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 68765d6674384d86b07dcf90be57f867, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680688500729 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting aa5e9cc7aac0401aa7a51e6e8995bf73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680688501382 2023-04-05 09:55:03,888 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0ca9f6e926d14282bd88092b1d57eaee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680688501382 2023-04-05 09:55:03,889 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ac7d2b8caa0240b89d3ea48a8387dd29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680688502541 2023-04-05 09:55:03,889 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting db59b6f991cd490cbd2a461e359e1b3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680688502541 2023-04-05 09:55:03,898 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#A#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:03,902 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#B#compaction#298 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:04,309 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/418590de312248e6ae40cb36ca8a2e35 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/418590de312248e6ae40cb36ca8a2e35 2023-04-05 09:55:04,312 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/9ec9b5d73fd842b885ab478a710ad559 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/9ec9b5d73fd842b885ab478a710ad559 2023-04-05 09:55:04,313 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/A of 8eb0aa9287d767302230619672527872 into 418590de312248e6ae40cb36ca8a2e35(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:04,313 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:04,313 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/A, priority=11, startTime=1680688503885; duration=0sec 2023-04-05 09:55:04,313 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:04,313 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:04,316 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:04,316 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 8eb0aa9287d767302230619672527872/C is initiating minor compaction (all files) 2023-04-05 09:55:04,316 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8eb0aa9287d767302230619672527872/C in TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:04,317 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d2cdd3ea95154d25adfce48c9235c32d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp, totalSize=60.4 K 2023-04-05 09:55:04,317 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d2cdd3ea95154d25adfce48c9235c32d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680688499952 2023-04-05 09:55:04,317 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/B of 8eb0aa9287d767302230619672527872 into 9ec9b5d73fd842b885ab478a710ad559(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:04,317 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:04,317 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/B, priority=11, startTime=1680688503886; duration=0sec 2023-04-05 09:55:04,317 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6481a43311da4adc9b11ce6ec84e8b16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680688500695 2023-04-05 09:55:04,317 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:04,318 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d6b136d1e64a4c4e86e8682d5a1948ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680688500729 2023-04-05 09:55:04,318 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ac04a8a0237a4ade9f18d56a68f22097, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680688501382 2023-04-05 09:55:04,318 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting bd173e5b02dc4212ab90bf899b22cb6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680688502541 2023-04-05 09:55:04,328 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8eb0aa9287d767302230619672527872#C#compaction#299 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:04,543 DEBUG [Thread-1643] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:62753 2023-04-05 09:55:04,543 DEBUG [Thread-1643] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:04,735 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/f148fd97c4f940c7b762472d57a9d07f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f148fd97c4f940c7b762472d57a9d07f 2023-04-05 09:55:04,739 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8eb0aa9287d767302230619672527872/C of 8eb0aa9287d767302230619672527872 into f148fd97c4f940c7b762472d57a9d07f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:04,739 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:04,739 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872., storeName=8eb0aa9287d767302230619672527872/C, priority=11, startTime=1680688503886; duration=0sec 2023-04-05 09:55:04,739 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:05,519 DEBUG [Thread-1635] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:62753 2023-04-05 09:55:05,519 DEBUG [Thread-1635] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:05,534 DEBUG [Thread-1641] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:62753 2023-04-05 09:55:05,534 DEBUG [Thread-1641] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:05,534 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 31 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5827 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5618 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5612 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5841 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 5592 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:55:05,535 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:55:05,535 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b4eae5 to 127.0.0.1:62753 2023-04-05 09:55:05,535 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:05,536 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:55:05,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:55:05,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:05,539 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688505539"}]},"ts":"1680688505539"} 2023-04-05 09:55:05,540 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:55:05,541 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:55:05,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, UNASSIGN}] 2023-04-05 09:55:05,543 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, UNASSIGN 2023-04-05 09:55:05,543 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8eb0aa9287d767302230619672527872, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:05,543 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688505543"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688505543"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688505543"}]},"ts":"1680688505543"} 2023-04-05 09:55:05,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:05,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8eb0aa9287d767302230619672527872 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8eb0aa9287d767302230619672527872, disabling compactions & flushes 2023-04-05 09:55:05,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. after waiting 0 ms 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:05,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8eb0aa9287d767302230619672527872 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=A 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=B 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8eb0aa9287d767302230619672527872, store=C 2023-04-05 09:55:05,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:06,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/7ca3cedfc4bd469fa4f2c44d42f68d96 2023-04-05 09:55:06,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:06,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e5658aea3c674fd5adbb9a1011f46efd 2023-04-05 09:55:06,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:06,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/879f1b390b5341218e2e6e806b7f344b 2023-04-05 09:55:06,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/A/7ca3cedfc4bd469fa4f2c44d42f68d96 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/7ca3cedfc4bd469fa4f2c44d42f68d96 2023-04-05 09:55:06,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/7ca3cedfc4bd469fa4f2c44d42f68d96, entries=150, sequenceid=561, filesize=11.9 K 2023-04-05 09:55:06,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/B/e5658aea3c674fd5adbb9a1011f46efd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e5658aea3c674fd5adbb9a1011f46efd 2023-04-05 09:55:06,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e5658aea3c674fd5adbb9a1011f46efd, entries=150, sequenceid=561, filesize=11.9 K 2023-04-05 09:55:06,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/.tmp/C/879f1b390b5341218e2e6e806b7f344b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/879f1b390b5341218e2e6e806b7f344b 2023-04-05 09:55:06,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/879f1b390b5341218e2e6e806b7f344b, entries=150, sequenceid=561, filesize=11.9 K 2023-04-05 09:55:06,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 8eb0aa9287d767302230619672527872 in 1239ms, sequenceid=561, compaction requested=false 2023-04-05 09:55:06,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/8a4d3c35bc2547ecbb11ca59759ffbfe, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/901105d6bfc34dfab1f4f449aad10cd4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/3e9551e60d2c42a294715e4f389f7e3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c1d09d3026c2436b934b59ae0b3de561, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/92a7401fbe5a40b7a375cd62c736d0ff, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4ae04402e7f7449094cf3d53550c90d0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/96e9bb246ab041188e0ec421bd343892, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/50c82b39f8c74f0885f80ccc7a6584e7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29] to archive 2023-04-05 09:55:06,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:06,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/dc6beea24f7848e5a29f565f15bee1f9 2023-04-05 09:55:06,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c9b9e70c7d64444f90dcf145e8e504e6 2023-04-05 09:55:06,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/8a4d3c35bc2547ecbb11ca59759ffbfe to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/8a4d3c35bc2547ecbb11ca59759ffbfe 2023-04-05 09:55:06,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1dfe290a86364875af4401507673d8a9 2023-04-05 09:55:06,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/e28ace1e4c78475b8e7c1dcc1eb613d5 2023-04-05 09:55:06,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1855a1bac02148e58ace9b1c0d0ec257 2023-04-05 09:55:06,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/901105d6bfc34dfab1f4f449aad10cd4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/901105d6bfc34dfab1f4f449aad10cd4 2023-04-05 09:55:06,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1ba0596d93d9455ca4719df0951a2b8b 2023-04-05 09:55:06,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/a53c50ccc122498881441755a0fca4f5 2023-04-05 09:55:06,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/3e9551e60d2c42a294715e4f389f7e3d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/3e9551e60d2c42a294715e4f389f7e3d 2023-04-05 09:55:06,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/36dd1ef54660475db0a168654d87eeac 2023-04-05 09:55:06,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c1d09d3026c2436b934b59ae0b3de561 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/c1d09d3026c2436b934b59ae0b3de561 2023-04-05 09:55:06,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9307dbcf70e447709bfaded7cd042c1a 2023-04-05 09:55:06,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f51c5da9af9342cda1b3558dad5911be 2023-04-05 09:55:06,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/fd4193c2f8194d908512c62b31b5cf43 2023-04-05 09:55:06,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/6372b1b14eba41c487476de208f94b90 2023-04-05 09:55:06,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/92a7401fbe5a40b7a375cd62c736d0ff to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/92a7401fbe5a40b7a375cd62c736d0ff 2023-04-05 09:55:06,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/f068f02061314e329aa22ae4c44ea7f4 2023-04-05 09:55:06,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4ae04402e7f7449094cf3d53550c90d0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4ae04402e7f7449094cf3d53550c90d0 2023-04-05 09:55:06,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/536ea165602b4ff6a0454842d29f767b 2023-04-05 09:55:06,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/d4567239cbd141e08a3e4150ce500a3d 2023-04-05 09:55:06,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/31ded033f973466fa96adb2e14769f54 2023-04-05 09:55:06,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/4837009ed2c3485087270c46c97b22cc 2023-04-05 09:55:06,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/96e9bb246ab041188e0ec421bd343892 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/96e9bb246ab041188e0ec421bd343892 2023-04-05 09:55:06,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/5bfc682a068c4e0fb785de404a7c237f 2023-04-05 09:55:06,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/138b5780d974494cbd2e1ebc1c04353d 2023-04-05 09:55:06,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/9c9cee466861475ba63ac42a51367eb3 2023-04-05 09:55:06,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/18c036564cd24c12bfd71a0a1a754694 2023-04-05 09:55:06,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/50c82b39f8c74f0885f80ccc7a6584e7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/50c82b39f8c74f0885f80ccc7a6584e7 2023-04-05 09:55:06,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/2fd675a4437346259857822deab8e7c4 2023-04-05 09:55:06,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/1a4ba3e233ce4b6b94d0f1dbb1c20801 2023-04-05 09:55:06,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/aa5e9cc7aac0401aa7a51e6e8995bf73 2023-04-05 09:55:06,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/ac7d2b8caa0240b89d3ea48a8387dd29 2023-04-05 09:55:06,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/c405fbbcc1184fa19eb157095a5e7835, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/21e3fc71dfb04ad9921e490821abb6dd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ca64b1769cce4319b758ddba53850599, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0878b32962954924a730bf71a84d4097, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/cb2fc4a69a7d417296b69342e2fbc91b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e7ee140cd78f42639e4f28ea38adeb15, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/037d5626719b43f8828b980390be7313, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0162232909ea4fd497ad43ea04b2167d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c] to archive 2023-04-05 09:55:06,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:07,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/f28dfcccf16446449eb29c76bc5e1a34 2023-04-05 09:55:07,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5996f4881ea7494da5501c76439d71e1 2023-04-05 09:55:07,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/c405fbbcc1184fa19eb157095a5e7835 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/c405fbbcc1184fa19eb157095a5e7835 2023-04-05 09:55:07,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/d46fc3f848d643dabdfd6091a9464629 2023-04-05 09:55:07,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/60bdf947ab50435ba250e6a6eef5bd0b 2023-04-05 09:55:07,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/21e3fc71dfb04ad9921e490821abb6dd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/21e3fc71dfb04ad9921e490821abb6dd 2023-04-05 09:55:07,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/eff232cacb8b4987a7fe89201c4813ff 2023-04-05 09:55:07,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db8d424c897c45248177124adddb0aed 2023-04-05 09:55:07,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ca64b1769cce4319b758ddba53850599 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ca64b1769cce4319b758ddba53850599 2023-04-05 09:55:07,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/982ab2d6e1a346e986403da79bf46250 2023-04-05 09:55:07,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/7b5983fc260a49758280bc9b16c87e26 2023-04-05 09:55:07,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0878b32962954924a730bf71a84d4097 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0878b32962954924a730bf71a84d4097 2023-04-05 09:55:07,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/33e14f0dfe8e46e1b8ae8efeb7e3fece 2023-04-05 09:55:07,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/029114b0246d4e7c8ec2c4f2fa97bf6c 2023-04-05 09:55:07,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/ee327e23cdbc413482f5326dd0e70bb2 2023-04-05 09:55:07,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/cb2fc4a69a7d417296b69342e2fbc91b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/cb2fc4a69a7d417296b69342e2fbc91b 2023-04-05 09:55:07,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/83ed4c77eba945a6b166085a7f9f343e 2023-04-05 09:55:07,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e4a0730ee48a4742b04c4ccc5243b6f5 2023-04-05 09:55:07,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e7ee140cd78f42639e4f28ea38adeb15 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e7ee140cd78f42639e4f28ea38adeb15 2023-04-05 09:55:07,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/81aacafe59bd46ccaf2fd762e7e5589c 2023-04-05 09:55:07,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/02d2a755f04746818d12d6a2b39654fd 2023-04-05 09:55:07,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5bfff9b1d7f14ec29930f97c6663aa3d 2023-04-05 09:55:07,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/037d5626719b43f8828b980390be7313 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/037d5626719b43f8828b980390be7313 2023-04-05 09:55:07,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/91c5314d226046ffb4820e9017f045ef 2023-04-05 09:55:07,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/5f3714d2848447a0a094954b760f8a83 2023-04-05 09:55:07,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e16f351245e645da8cbcb6f67194f3da 2023-04-05 09:55:07,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/a0557980a5b64251b7362f11361d1d17 2023-04-05 09:55:07,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0162232909ea4fd497ad43ea04b2167d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0162232909ea4fd497ad43ea04b2167d 2023-04-05 09:55:07,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/45799397adca437a8b7f84b2c3a2e885 2023-04-05 09:55:07,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/6c727a11b7474b9588d34b08af815871 2023-04-05 09:55:07,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/68765d6674384d86b07dcf90be57f867 2023-04-05 09:55:07,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/0ca9f6e926d14282bd88092b1d57eaee 2023-04-05 09:55:07,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/db59b6f991cd490cbd2a461e359e1b3c 2023-04-05 09:55:07,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ec73b2f970f64fb1bab658441334ea0a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/5d8cbc5ef39949968c5c45c9d22cebd1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/215e1446c9e54625a1b781e60003d995, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cf82b9bd1f5a46a1bde44cf49ff948e3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cedb077e8d0a42f9accf92ad890bcc8a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8f4b16ccb9214831896bdb558021b338, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b3259f3a42cd42c7bbbf71c39821fe79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d2cdd3ea95154d25adfce48c9235c32d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c] to archive 2023-04-05 09:55:07,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:07,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/59e4679671ff4876ab6d36f77d116f50 2023-04-05 09:55:07,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/7f87be579f414c3b8d826e04e3a4b4ee 2023-04-05 09:55:07,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ec73b2f970f64fb1bab658441334ea0a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ec73b2f970f64fb1bab658441334ea0a 2023-04-05 09:55:07,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b5d65ebcad9741398078922d02f400e9 2023-04-05 09:55:07,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8de6e42d12334a9a84b7c1a6bb6c9359 2023-04-05 09:55:07,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/5d8cbc5ef39949968c5c45c9d22cebd1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/5d8cbc5ef39949968c5c45c9d22cebd1 2023-04-05 09:55:07,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f0c2bf7aed8d4717b2e32f6e2fe2c52e 2023-04-05 09:55:07,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8650e6fa50114466af845bd473a01798 2023-04-05 09:55:07,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/215e1446c9e54625a1b781e60003d995 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/215e1446c9e54625a1b781e60003d995 2023-04-05 09:55:07,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8631334a4cc343d0bb64a7938dc7542e 2023-04-05 09:55:07,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/85664e7de51841c7bde69230cd797507 2023-04-05 09:55:07,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cf82b9bd1f5a46a1bde44cf49ff948e3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cf82b9bd1f5a46a1bde44cf49ff948e3 2023-04-05 09:55:07,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a90b1112076a47b6aaac3e9a05630d54 2023-04-05 09:55:07,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4e366e4714ab4fc1bbb59004e47c371c 2023-04-05 09:55:07,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/79e2ce296cd3437aba9fd006d6559113 2023-04-05 09:55:07,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cedb077e8d0a42f9accf92ad890bcc8a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/cedb077e8d0a42f9accf92ad890bcc8a 2023-04-05 09:55:07,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/78463f7774284af299c8416ebbf80b54 2023-04-05 09:55:07,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/82e818aceac24081be000941624f13d1 2023-04-05 09:55:07,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8f4b16ccb9214831896bdb558021b338 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8f4b16ccb9214831896bdb558021b338 2023-04-05 09:55:07,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a1c019995b3e4de4b4d37caf37d50102 2023-04-05 09:55:07,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/fe87fffbc5404808ba9ba8720566d605 2023-04-05 09:55:07,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/8dd89ab7b4124e998d3f86a626c37946 2023-04-05 09:55:07,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b3259f3a42cd42c7bbbf71c39821fe79 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b3259f3a42cd42c7bbbf71c39821fe79 2023-04-05 09:55:07,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/a412177e378d4c5fb7adf864479fd98c 2023-04-05 09:55:07,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6c575db4ec2d4ee9890e7ba21e835e66 2023-04-05 09:55:07,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/4c0f4e831b7b48369c9c05eb2fd89644 2023-04-05 09:55:07,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/216a5960343d40a899ac2e0b961d2f8d 2023-04-05 09:55:07,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d2cdd3ea95154d25adfce48c9235c32d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d2cdd3ea95154d25adfce48c9235c32d 2023-04-05 09:55:07,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/b1594c8541ac4c94ba1f1105f25a319f 2023-04-05 09:55:07,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/6481a43311da4adc9b11ce6ec84e8b16 2023-04-05 09:55:07,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/d6b136d1e64a4c4e86e8682d5a1948ae 2023-04-05 09:55:07,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/ac04a8a0237a4ade9f18d56a68f22097 2023-04-05 09:55:07,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/bd173e5b02dc4212ab90bf899b22cb6c 2023-04-05 09:55:07,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/recovered.edits/564.seqid, newMaxSeqId=564, maxSeqId=1 2023-04-05 09:55:07,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872. 2023-04-05 09:55:07,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8eb0aa9287d767302230619672527872: 2023-04-05 09:55:07,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8eb0aa9287d767302230619672527872 2023-04-05 09:55:07,070 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8eb0aa9287d767302230619672527872, regionState=CLOSED 2023-04-05 09:55:07,070 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688507070"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688507070"}]},"ts":"1680688507070"} 2023-04-05 09:55:07,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-05 09:55:07,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 8eb0aa9287d767302230619672527872, server=jenkins-hbase4.apache.org,46827,1680688416991 in 1.5270 sec 2023-04-05 09:55:07,073 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-05 09:55:07,073 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8eb0aa9287d767302230619672527872, UNASSIGN in 1.5310 sec 2023-04-05 09:55:07,074 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688507074"}]},"ts":"1680688507074"} 2023-04-05 09:55:07,075 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:55:07,076 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:55:07,077 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5400 sec 2023-04-05 09:55:07,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 09:55:07,643 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-05 09:55:07,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:55:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,645 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 09:55:07,646 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,650 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:55:07,652 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/recovered.edits] 2023-04-05 09:55:07,654 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/418590de312248e6ae40cb36ca8a2e35 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/418590de312248e6ae40cb36ca8a2e35 2023-04-05 09:55:07,655 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/7ca3cedfc4bd469fa4f2c44d42f68d96 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/A/7ca3cedfc4bd469fa4f2c44d42f68d96 2023-04-05 09:55:07,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/9ec9b5d73fd842b885ab478a710ad559 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/9ec9b5d73fd842b885ab478a710ad559 2023-04-05 09:55:07,658 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e5658aea3c674fd5adbb9a1011f46efd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/B/e5658aea3c674fd5adbb9a1011f46efd 2023-04-05 09:55:07,660 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/879f1b390b5341218e2e6e806b7f344b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/879f1b390b5341218e2e6e806b7f344b 2023-04-05 09:55:07,661 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f148fd97c4f940c7b762472d57a9d07f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/C/f148fd97c4f940c7b762472d57a9d07f 2023-04-05 09:55:07,663 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/recovered.edits/564.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872/recovered.edits/564.seqid 2023-04-05 09:55:07,663 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/8eb0aa9287d767302230619672527872 2023-04-05 09:55:07,663 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:55:07,665 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,666 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:55:07,668 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:55:07,668 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,668 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:55:07,669 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688507669"}]},"ts":"9223372036854775807"} 2023-04-05 09:55:07,670 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:55:07,670 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8eb0aa9287d767302230619672527872, NAME => 'TestAcidGuarantees,,1680688480435.8eb0aa9287d767302230619672527872.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:55:07,670 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:55:07,670 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688507670"}]},"ts":"9223372036854775807"} 2023-04-05 09:55:07,671 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:55:07,673 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,674 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-05 09:55:07,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 09:55:07,747 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-05 09:55:07,758 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=552 (was 539) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 426), ProcessCount=169 (was 169), AvailableMemoryMB=1024 (was 1123) 2023-04-05 09:55:07,769 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=169, AvailableMemoryMB=1023 2023-04-05 09:55:07,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:55:07,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:55:07,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:07,773 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:55:07,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-05 09:55:07,773 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:55:07,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 09:55:07,774 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:07,775 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb empty. 2023-04-05 09:55:07,775 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:07,775 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:55:07,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 09:55:08,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 09:55:08,182 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:55:08,183 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fb59cf4f6165326988fc5be4f32ae6fb, NAME => 'TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:55:08,189 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:55:08,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fb59cf4f6165326988fc5be4f32ae6fb, disabling compactions & flushes 2023-04-05 09:55:08,190 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. after waiting 0 ms 2023-04-05 09:55:08,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,190 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:08,192 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:55:08,193 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688508193"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688508193"}]},"ts":"1680688508193"} 2023-04-05 09:55:08,194 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:55:08,194 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:55:08,195 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688508194"}]},"ts":"1680688508194"} 2023-04-05 09:55:08,195 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:55:08,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, ASSIGN}] 2023-04-05 09:55:08,199 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, ASSIGN 2023-04-05 09:55:08,200 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:55:08,351 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:08,351 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688508351"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688508351"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688508351"}]},"ts":"1680688508351"} 2023-04-05 09:55:08,352 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:08,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 09:55:08,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb59cf4f6165326988fc5be4f32ae6fb, NAME => 'TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:55:08,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:55:08,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,508 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,509 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A 2023-04-05 09:55:08,509 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A 2023-04-05 09:55:08,509 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:08,510 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName A 2023-04-05 09:55:08,510 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:08,510 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,511 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B 2023-04-05 09:55:08,511 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B 2023-04-05 09:55:08,511 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:08,511 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName B 2023-04-05 09:55:08,512 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:08,512 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,513 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C 2023-04-05 09:55:08,513 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C 2023-04-05 09:55:08,513 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:08,513 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName C 2023-04-05 09:55:08,513 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:08,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:55:08,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:08,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:55:08,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb59cf4f6165326988fc5be4f32ae6fb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67967202, jitterRate=0.01279023289680481}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:55:08,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:08,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., pid=37, masterSystemTime=1680688508503 2023-04-05 09:55:08,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:08,521 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:08,521 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688508520"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688508520"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688508520"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688508520"}]},"ts":"1680688508520"} 2023-04-05 09:55:08,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-05 09:55:08,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 in 169 msec 2023-04-05 09:55:08,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-05 09:55:08,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, ASSIGN in 325 msec 2023-04-05 09:55:08,524 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:55:08,524 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688508524"}]},"ts":"1680688508524"} 2023-04-05 09:55:08,525 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:55:08,527 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:55:08,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 756 msec 2023-04-05 09:55:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 09:55:08,877 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-05 09:55:08,877 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:08,883 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:08,884 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:08,886 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:08,887 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:55:08,888 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:55:08,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:55:08,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:55:08,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:09,299 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:55:09,300 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:55:09,300 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:55:09,302 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 09:55:09,305 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, REOPEN/MOVE}] 2023-04-05 09:55:09,305 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, REOPEN/MOVE 2023-04-05 09:55:09,306 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,306 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688509306"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688509306"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688509306"}]},"ts":"1680688509306"} 2023-04-05 09:55:09,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:09,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb59cf4f6165326988fc5be4f32ae6fb, disabling compactions & flushes 2023-04-05 09:55:09,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. after waiting 0 ms 2023-04-05 09:55:09,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 09:55:09,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:09,463 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fb59cf4f6165326988fc5be4f32ae6fb to self. 2023-04-05 09:55:09,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,465 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=CLOSED 2023-04-05 09:55:09,465 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688509465"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688509465"}]},"ts":"1680688509465"} 2023-04-05 09:55:09,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-05 09:55:09,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 in 158 msec 2023-04-05 09:55:09,467 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=true 2023-04-05 09:55:09,617 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,618 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688509617"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688509617"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688509617"}]},"ts":"1680688509617"} 2023-04-05 09:55:09,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:09,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb59cf4f6165326988fc5be4f32ae6fb, NAME => 'TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:55:09,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:55:09,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,776 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,777 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A 2023-04-05 09:55:09,777 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A 2023-04-05 09:55:09,777 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:09,777 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName A 2023-04-05 09:55:09,779 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:09,779 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,780 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B 2023-04-05 09:55:09,780 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B 2023-04-05 09:55:09,780 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:09,780 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName B 2023-04-05 09:55:09,781 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:09,781 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,782 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C 2023-04-05 09:55:09,782 DEBUG [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C 2023-04-05 09:55:09,782 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:09,782 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb59cf4f6165326988fc5be4f32ae6fb columnFamilyName C 2023-04-05 09:55:09,783 INFO [StoreOpener-fb59cf4f6165326988fc5be4f32ae6fb-1] regionserver.HStore(310): Store=fb59cf4f6165326988fc5be4f32ae6fb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:09,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:55:09,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb59cf4f6165326988fc5be4f32ae6fb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71671416, jitterRate=0.06798732280731201}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:55:09,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:09,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., pid=42, masterSystemTime=1680688509770 2023-04-05 09:55:09,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,790 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,790 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688509790"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688509790"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688509790"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688509790"}]},"ts":"1680688509790"} 2023-04-05 09:55:09,792 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-05 09:55:09,792 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 in 172 msec 2023-04-05 09:55:09,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-05 09:55:09,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, REOPEN/MOVE in 487 msec 2023-04-05 09:55:09,795 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-05 09:55:09,795 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-05 09:55:09,797 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-05 09:55:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-05 09:55:09,798 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,802 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,803 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,806 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,806 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,809 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,809 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,813 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,813 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,816 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,817 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,820 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,820 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,823 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,823 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,829 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,829 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,833 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,833 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:09,836 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:09,840 DEBUG [hconnection-0x3318cba7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,841 DEBUG [hconnection-0x600b91ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,841 DEBUG [hconnection-0x1ee7999-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,841 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:09,843 DEBUG [hconnection-0x3cb9f896-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,843 DEBUG [hconnection-0x52100ecc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,843 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,843 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,844 DEBUG [hconnection-0x325bb8ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,844 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,845 DEBUG [hconnection-0x5eacec96-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,845 DEBUG [hconnection-0x760f2276-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:09,847 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,848 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:09,848 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,850 DEBUG [hconnection-0x51e12744-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,849 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,848 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:09,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:09,850 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:09,848 DEBUG [hconnection-0x64821d66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:09,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:09,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:09,857 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,857 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:09,858 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:09,858 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:09,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:09,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:09,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:09,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:09,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:09,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:09,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:09,867 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 09:55:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:09,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:09,868 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:09,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688569882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688569883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1680688509872, max timestamp=1680688509883Num uniques 859; 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1680688509872, max timestamp=1680688509883Num uniques 859; 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1680688509872, max timestamp=1680688509883Num uniques 859; 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-05 09:55:09,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-05 09:55:09,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688569897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688569898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688569901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688569987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:09,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:09,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688569991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688570003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688570004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688570009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688570189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688570195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688570207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688570207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688570217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,301 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405127a82a541cf4c8c98e568f6992052f9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405127a82a541cf4c8c98e568f6992052f9 2023-04-05 09:55:10,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688570496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688570500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688570513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688570513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:10,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688570523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:10,707 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/b2dde124e3e64f37821cc9c9205ab48c 2023-04-05 09:55:11,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:11,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688571003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:11,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688571006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:11,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688571018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:11,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688571019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:11,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688571029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,117 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/1575355354904c0ba47036a43783b088 2023-04-05 09:55:11,527 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/41177b76f20440ffa6b628994ae06a1c 2023-04-05 09:55:11,530 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/b2dde124e3e64f37821cc9c9205ab48c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c 2023-04-05 09:55:11,534 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c, entries=200, sequenceid=14, filesize=31.9 K 2023-04-05 09:55:11,534 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/1575355354904c0ba47036a43783b088 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088 2023-04-05 09:55:11,538 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 09:55:11,538 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/41177b76f20440ffa6b628994ae06a1c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c 2023-04-05 09:55:11,541 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 09:55:11,542 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for fb59cf4f6165326988fc5be4f32ae6fb in 1675ms, sequenceid=14, compaction requested=false 2023-04-05 09:55:11,542 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:11,542 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:11,545 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:11,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:11,545 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,545 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:11,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:11,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:11,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a35d9ba[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:11,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:11,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:11,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:11,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:11,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:11,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:11,552 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:11,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:11,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:11,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:11,552 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:11,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:11,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:11,554 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:11,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:11,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:11,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:11,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:11,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:11,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:11,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:11,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:11,560 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:11,560 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:11,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:11,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:11,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,660 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:11,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:11,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:11,663 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:11,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:11,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:11,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:11,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:11,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:11,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:11,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:11,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:11,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:11,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:11,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:11,667 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=147.60 KB heapSize=387.33 KB 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:11,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:12,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:12,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:12,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688572024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688572027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688572028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688572030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688572034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,083 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405307cb127e43c4837846c3a26dbe9168b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405307cb127e43c4837846c3a26dbe9168b 2023-04-05 09:55:12,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688572131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688572136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688572336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688572340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,490 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=45, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/e64e965070b741debf7d5920eb730f4f 2023-04-05 09:55:12,602 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:55:12,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688572642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:12,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688572644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:12,902 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/58212befc97c40f794a4d4333052f472 2023-04-05 09:55:13,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688573151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:13,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688573151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,313 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/547f8cdf59624f41b68b9afc22013d2f 2023-04-05 09:55:13,317 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/e64e965070b741debf7d5920eb730f4f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f 2023-04-05 09:55:13,320 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f, entries=150, sequenceid=45, filesize=25.1 K 2023-04-05 09:55:13,321 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/58212befc97c40f794a4d4333052f472 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472 2023-04-05 09:55:13,325 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472, entries=150, sequenceid=45, filesize=11.6 K 2023-04-05 09:55:13,326 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/547f8cdf59624f41b68b9afc22013d2f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f 2023-04-05 09:55:13,329 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f, entries=150, sequenceid=45, filesize=11.6 K 2023-04-05 09:55:13,329 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~387.28 KB/396576, currentSize=46.96 KB/48090 for fb59cf4f6165326988fc5be4f32ae6fb in 1662ms, sequenceid=45, compaction requested=false 2023-04-05 09:55:13,330 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:13,330 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:13,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:13,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:13,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:13,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:13,331 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:13,332 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,332 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5113cc1b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:13,333 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:13,333 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,334 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:13,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:13,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:13,334 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,334 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:13,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:13,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:13,342 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:13,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:13,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:13,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:13,342 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:13,343 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,343 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,343 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:13,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:13,344 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:13,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:13,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:13,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:13,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:13,348 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:13,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:13,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:13,348 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:13,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:13,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,448 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:13,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:13,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:13,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:13,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:13,452 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:13,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:13,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:13,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:13,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:13,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:13,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:13,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:13,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:13,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:13,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:13,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:13,456 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:13,456 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:13,457 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:13,457 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:13,869 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405aa9f52923805405b820accdca92b9fab to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aa9f52923805405b820accdca92b9fab 2023-04-05 09:55:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:14,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:14,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688514064, max timestamp=1680688514078Num uniques 859; 2023-04-05 09:55:14,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688514064, max timestamp=1680688514078Num uniques 859; 2023-04-05 09:55:14,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688514064, max timestamp=1680688514078Num uniques 859; 2023-04-05 09:55:14,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:14,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:14,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:14,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688574136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688574136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688574136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688574157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688574162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688574249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688574249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688574250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,274 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=55, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/ff461c3275ec4f80a3613abd69150131 2023-04-05 09:55:14,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688574454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688574458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688574461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,686 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/9cd97f08bca94ad0b2f0a96cb847cc2c 2023-04-05 09:55:14,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688574755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688574778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688574778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,095 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0 2023-04-05 09:55:15,099 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/ff461c3275ec4f80a3613abd69150131 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131 2023-04-05 09:55:15,102 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131, entries=150, sequenceid=55, filesize=25.1 K 2023-04-05 09:55:15,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/9cd97f08bca94ad0b2f0a96cb847cc2c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c 2023-04-05 09:55:15,106 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c, entries=150, sequenceid=55, filesize=11.6 K 2023-04-05 09:55:15,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0 2023-04-05 09:55:15,110 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0, entries=150, sequenceid=55, filesize=11.6 K 2023-04-05 09:55:15,111 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for fb59cf4f6165326988fc5be4f32ae6fb in 1655ms, sequenceid=55, compaction requested=true 2023-04-05 09:55:15,111 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:15,111 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:15,112 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,113 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,113 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8e9f26[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:15,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:15,114 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:15,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:15,116 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:15,120 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:15,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:15,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:15,120 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:15,120 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:15,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:15,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:15,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,125 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,127 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,127 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:15,127 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:15,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:15,127 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:15,127 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:15,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:15,128 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:15,128 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:15,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:15,228 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:15,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:15,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:15,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:15,252 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:15,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:15,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:15,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:15,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,253 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:15,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:15,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:15,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:15,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:15,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:15,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:15,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:15,261 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:15,261 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:15,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:15,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:15,288 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040585dfcd784f1f4c9d809c52e5f5f43032 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040585dfcd784f1f4c9d809c52e5f5f43032 2023-04-05 09:55:15,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688575283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,294 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=92, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/67dfcc12c8a348cc9c674ccc43cb8671 2023-04-05 09:55:15,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688575289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688575292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688575393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688575401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688575401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688575600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688575611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688575611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,743 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/16c89e67337d483383707517f352b5b8 2023-04-05 09:55:15,758 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/974145d2285a47a38308ec0b6c8ac102 2023-04-05 09:55:15,762 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/67dfcc12c8a348cc9c674ccc43cb8671 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671 2023-04-05 09:55:15,766 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671, entries=150, sequenceid=92, filesize=25.1 K 2023-04-05 09:55:15,767 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/16c89e67337d483383707517f352b5b8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8 2023-04-05 09:55:15,770 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 09:55:15,771 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/974145d2285a47a38308ec0b6c8ac102 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102 2023-04-05 09:55:15,773 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 09:55:15,774 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for fb59cf4f6165326988fc5be4f32ae6fb in 513ms, sequenceid=92, compaction requested=true 2023-04-05 09:55:15,774 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:15,774 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:15,774 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:15,774 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:15,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,776 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:15,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bba227d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:15,779 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:15,779 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,780 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:15,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:15,780 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:15,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:15,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:15,784 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:15,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:15,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:15,785 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:15,785 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:15,787 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,787 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,787 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:15,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:15,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:15,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:15,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:15,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:15,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,790 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:15,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:15,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:15,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:15,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:15,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:15,791 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:15,791 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:15,891 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:15,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:15,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:15,900 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:15,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:15,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:15,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:15,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:15,901 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:15,902 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,902 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:15,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:15,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:15,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:15,903 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:15,903 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:15,904 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:15,904 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:15,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688515940, max timestamp=1680688515949Num uniques 859; 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688515940, max timestamp=1680688515949Num uniques 859; 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688515940, max timestamp=1680688515949Num uniques 859; 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:15,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:15,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688575983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688575986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688575989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688576092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688576092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688576094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688576167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688576169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,176 DEBUG [Thread-2356] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:16,176 DEBUG [Thread-2348] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:16,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688576300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688576300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688576301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,322 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405fbf0f04b99424c5da9ce9c6b58536f51 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fbf0f04b99424c5da9ce9c6b58536f51 2023-04-05 09:55:16,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688576605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688576606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:16,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688576607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:16,730 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/f28887de0ad64a619e198346393b49e0 2023-04-05 09:55:16,742 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/532ef50fab45442a9fe8fae850370d73 2023-04-05 09:55:17,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:17,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688577115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:17,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688577116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:17,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688577117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,163 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/32dd5c7f02764667941b137644bc56cd 2023-04-05 09:55:17,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/f28887de0ad64a619e198346393b49e0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0 2023-04-05 09:55:17,181 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0, entries=150, sequenceid=101, filesize=25.1 K 2023-04-05 09:55:17,182 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/532ef50fab45442a9fe8fae850370d73 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73 2023-04-05 09:55:17,186 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73, entries=150, sequenceid=101, filesize=11.6 K 2023-04-05 09:55:17,186 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/32dd5c7f02764667941b137644bc56cd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd 2023-04-05 09:55:17,193 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd, entries=150, sequenceid=101, filesize=11.6 K 2023-04-05 09:55:17,194 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for fb59cf4f6165326988fc5be4f32ae6fb in 1291ms, sequenceid=101, compaction requested=true 2023-04-05 09:55:17,194 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:17,194 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,195 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:17,195 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:17,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,196 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:17,196 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,197 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ff6cab9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:17,198 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:17,198 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,199 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:17,200 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,200 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:17,202 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:17,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:17,205 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:17,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:17,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:17,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:17,205 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:17,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:17,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:17,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:17,209 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,211 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:17,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:17,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:17,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:17,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:17,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:17,216 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:17,216 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:17,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:17,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:17,317 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:17,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:17,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:17,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:17,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:17,320 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:17,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:17,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:17,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:17,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:17,322 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:17,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:17,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:17,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:17,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:17,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:17,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:17,323 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:17,324 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:17,324 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:17,739 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b34472e6c3024081acd4ab7c2d24af5f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b34472e6c3024081acd4ab7c2d24af5f 2023-04-05 09:55:17,745 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/2fc7202cae6c4ad684cd1798b8c9c0bf 2023-04-05 09:55:18,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:18,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688578142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688578143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688578143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,154 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/b2f0046b3c1b4b81b4671879a2859f8a 2023-04-05 09:55:18,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688578247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688578251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688578253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688578454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688578459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688578460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,565 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/35a03abb430e439fb9fdafae233a7a79 2023-04-05 09:55:18,570 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/2fc7202cae6c4ad684cd1798b8c9c0bf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf 2023-04-05 09:55:18,575 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf, entries=150, sequenceid=138, filesize=25.3 K 2023-04-05 09:55:18,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/b2f0046b3c1b4b81b4671879a2859f8a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a 2023-04-05 09:55:18,581 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a, entries=150, sequenceid=138, filesize=11.8 K 2023-04-05 09:55:18,581 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/35a03abb430e439fb9fdafae233a7a79 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79 2023-04-05 09:55:18,586 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79, entries=150, sequenceid=138, filesize=11.8 K 2023-04-05 09:55:18,587 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for fb59cf4f6165326988fc5be4f32ae6fb in 1263ms, sequenceid=138, compaction requested=true 2023-04-05 09:55:18,587 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:18,587 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,588 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:18,588 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:18,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:18,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:18,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:18,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30caa248[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:18,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:18,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,597 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:18,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:18,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:18,597 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,597 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:18,598 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:18,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:18,602 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:18,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:18,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:18,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:18,602 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:18,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,604 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:18,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:18,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:18,620 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:18,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:18,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:18,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:18,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:18,620 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:18,620 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:18,621 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:18,721 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:18,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:18,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:18,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:18,732 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:18,732 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:18,732 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:18,732 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:18,732 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:18,733 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:18,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:18,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:18,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:18,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:18,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:18,736 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:18,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:18,752 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052f8bfa6df8714e25889425d135f2e3e0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052f8bfa6df8714e25889425d135f2e3e0 2023-04-05 09:55:18,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:18,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:18,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688518815, max timestamp=1680688518845Num uniques 859; 2023-04-05 09:55:18,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:18,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688518815, max timestamp=1680688518845Num uniques 859; 2023-04-05 09:55:18,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:18,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688518815, max timestamp=1680688518845Num uniques 859; 2023-04-05 09:55:18,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:18,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688578916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688578917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:18,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688578923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:19,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1680688579024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:19,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41356 deadline: 1680688579027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:19,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688579033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,157 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/375cbdccd33941c38e3d59795e41c845 2023-04-05 09:55:19,170 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/c46bd4b495284562bb04ccb8c28bef4b 2023-04-05 09:55:19,186 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/99662e5282d54f378a6c91c9d04b3fb7 2023-04-05 09:55:19,190 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/375cbdccd33941c38e3d59795e41c845 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845 2023-04-05 09:55:19,195 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845, entries=100, sequenceid=147, filesize=18.5 K 2023-04-05 09:55:19,196 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/c46bd4b495284562bb04ccb8c28bef4b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b 2023-04-05 09:55:19,201 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b, entries=100, sequenceid=147, filesize=9.4 K 2023-04-05 09:55:19,201 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/99662e5282d54f378a6c91c9d04b3fb7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7 2023-04-05 09:55:19,205 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7, entries=100, sequenceid=147, filesize=9.4 K 2023-04-05 09:55:19,205 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for fb59cf4f6165326988fc5be4f32ae6fb in 469ms, sequenceid=147, compaction requested=true 2023-04-05 09:55:19,206 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:19,206 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,207 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae77cd8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,212 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,213 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,216 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,216 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,219 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,219 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,219 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,221 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,225 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,225 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,226 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688518915, max timestamp=1680688519239Num uniques 1068; 2023-04-05 09:55:19,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688518915, max timestamp=1680688519239Num uniques 1068; 2023-04-05 09:55:19,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:55:19,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688518915, max timestamp=1680688519239Num uniques 1068; 2023-04-05 09:55:19,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:55:19,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 09:55:19,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:19,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:19,326 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688519295, max timestamp=1680688519326Num uniques 859; 2023-04-05 09:55:19,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:19,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688519295, max timestamp=1680688519326Num uniques 859; 2023-04-05 09:55:19,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:19,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688519295, max timestamp=1680688519326Num uniques 859; 2023-04-05 09:55:19,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:19,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,341 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,343 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,343 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,344 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,344 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,345 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,345 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,346 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,346 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70541233[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,351 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,352 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,352 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,356 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,356 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,359 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,364 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,364 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,364 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680688519384, max timestamp=1680688519413Num uniques 1068; 2023-04-05 09:55:19,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680688519384, max timestamp=1680688519413Num uniques 1068; 2023-04-05 09:55:19,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 09:55:19,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 09:55:19,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680688519384, max timestamp=1680688519413Num uniques 1068; 2023-04-05 09:55:19,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 09:55:19,465 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,468 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,471 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,471 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,473 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,473 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,473 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,473 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,474 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688519456, max timestamp=1680688519475Num uniques 1127; 2023-04-05 09:55:19,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-05 09:55:19,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688519456, max timestamp=1680688519475Num uniques 1127; 2023-04-05 09:55:19,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-05 09:55:19,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680688519456, max timestamp=1680688519475Num uniques 1127; 2023-04-05 09:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-05 09:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393ebd82[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,481 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,483 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,483 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,487 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,488 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,489 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,489 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,490 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,490 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,490 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688519505, max timestamp=1680688519550Num uniques 1036; 2023-04-05 09:55:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-05 09:55:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688519505, max timestamp=1680688519550Num uniques 1036; 2023-04-05 09:55:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680688519505, max timestamp=1680688519550Num uniques 1036; 2023-04-05 09:55:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-05 09:55:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-05 09:55:19,590 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,593 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,595 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,596 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,597 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,597 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,597 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,597 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,598 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c83900f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,601 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,606 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,606 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,610 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,610 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,611 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:19,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41422 deadline: 1680688579614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688519605, max timestamp=1680688519629Num uniques 1145; 2023-04-05 09:55:19,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-05 09:55:19,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688519605, max timestamp=1680688519629Num uniques 1145; 2023-04-05 09:55:19,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680688519605, max timestamp=1680688519629Num uniques 1145; 2023-04-05 09:55:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-05 09:55:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-05 09:55:19,656 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040532bb2a8573f34ff48048756d45a1fd9e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040532bb2a8573f34ff48048756d45a1fd9e 2023-04-05 09:55:19,662 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/d9dda9900d7c4aaa8c5b01fa29799ce7 2023-04-05 09:55:19,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/65cc924fee724f0595bbcfc9505e5875 2023-04-05 09:55:19,711 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,715 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,715 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,717 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,734 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,734 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,734 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,734 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd336fb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,735 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,735 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,736 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,739 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680688519724, max timestamp=1680688519741Num uniques 1104; 2023-04-05 09:55:19,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-05 09:55:19,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680688519724, max timestamp=1680688519741Num uniques 1104; 2023-04-05 09:55:19,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-05 09:55:19,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680688519724, max timestamp=1680688519741Num uniques 1104; 2023-04-05 09:55:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-05 09:55:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,746 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,746 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,748 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,748 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,751 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,753 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,754 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,753 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680688519779, max timestamp=1680688519819Num uniques 1063; 2023-04-05 09:55:19,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-05 09:55:19,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680688519779, max timestamp=1680688519819Num uniques 1063; 2023-04-05 09:55:19,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-05 09:55:19,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680688519779, max timestamp=1680688519819Num uniques 1063; 2023-04-05 09:55:19,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-05 09:55:19,854 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,859 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,864 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,866 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,867 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,868 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc668f8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:19,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:19,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:19,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:19,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:19,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:19,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:19,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:19,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:19,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:19,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:19,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:19,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:19,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,882 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:19,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,882 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:19,883 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:19,883 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:19,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688519853, max timestamp=1680688519914Num uniques 1072; 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688519853, max timestamp=1680688519914Num uniques 1072; 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680688519853, max timestamp=1680688519914Num uniques 1072; 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-05 09:55:19,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-05 09:55:19,983 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:19,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:19,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:19,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:19,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:19,988 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:19,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:19,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:19,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,989 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:19,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:19,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:19,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,990 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:19,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:19,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:19,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:19,991 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,991 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:19,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:19,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:19,997 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:19,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,000 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11581c6f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,002 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,002 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,003 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,005 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,005 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,005 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,005 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,008 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,008 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,008 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,008 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,008 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680688519976, max timestamp=1680688520011Num uniques 1081; 2023-04-05 09:55:20,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-05 09:55:20,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680688519976, max timestamp=1680688520011Num uniques 1081; 2023-04-05 09:55:20,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-05 09:55:20,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680688519976, max timestamp=1680688520011Num uniques 1081; 2023-04-05 09:55:20,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-05 09:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,014 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,018 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,018 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:20,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680688520060, max timestamp=1680688520084Num uniques 1090; 2023-04-05 09:55:20,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-05 09:55:20,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680688520060, max timestamp=1680688520084Num uniques 1090; 2023-04-05 09:55:20,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-05 09:55:20,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680688520060, max timestamp=1680688520084Num uniques 1090; 2023-04-05 09:55:20,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-05 09:55:20,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/09b2afbf182f4674be6eb0e1d2fbad99 2023-04-05 09:55:20,118 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/d9dda9900d7c4aaa8c5b01fa29799ce7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7 2023-04-05 09:55:20,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,130 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,134 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7, entries=200, sequenceid=185, filesize=32.1 K 2023-04-05 09:55:20,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/65cc924fee724f0595bbcfc9505e5875 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875 2023-04-05 09:55:20,135 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875, entries=150, sequenceid=185, filesize=11.8 K 2023-04-05 09:55:20,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/09b2afbf182f4674be6eb0e1d2fbad99 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99 2023-04-05 09:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519bf5a3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,149 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,150 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,150 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,150 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99, entries=150, sequenceid=185, filesize=11.8 K 2023-04-05 09:55:20,154 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=116.29 KB/119080 for fb59cf4f6165326988fc5be4f32ae6fb in 913ms, sequenceid=185, compaction requested=true 2023-04-05 09:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:20,156 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 09:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,157 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,157 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,157 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:20,158 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,158 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,158 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 09:55:20,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:20,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:20,160 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,159 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,159 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,163 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,163 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 93144 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 09:55:20,163 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/B is initiating minor compaction (all files) 2023-04-05 09:55:20,163 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/B in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,163 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=91.0 K 2023-04-05 09:55:20,164 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 213350 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 09:55:20,164 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/A is initiating minor compaction (all files) 2023-04-05 09:55:20,164 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/A in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,164 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=208.3 K 2023-04-05 09:55:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,164 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1575355354904c0ba47036a43783b088, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688509854 2023-04-05 09:55:20,164 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b2dde124e3e64f37821cc9c9205ab48c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688509854 2023-04-05 09:55:20,165 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 58212befc97c40f794a4d4333052f472, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680688509896 2023-04-05 09:55:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,165 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e64e965070b741debf7d5920eb730f4f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680688509896 2023-04-05 09:55:20,165 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9cd97f08bca94ad0b2f0a96cb847cc2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680688512023 2023-04-05 09:55:20,166 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 16c89e67337d483383707517f352b5b8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680688514112 2023-04-05 09:55:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,166 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 532ef50fab45442a9fe8fae850370d73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688515265 2023-04-05 09:55:20,167 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ff461c3275ec4f80a3613abd69150131, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680688512023 2023-04-05 09:55:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,168 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 67dfcc12c8a348cc9c674ccc43cb8671, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680688514112 2023-04-05 09:55:20,168 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b2f0046b3c1b4b81b4671879a2859f8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680688515982 2023-04-05 09:55:20,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,169 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f28887de0ad64a619e198346393b49e0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688515265 2023-04-05 09:55:20,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,170 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:20,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=129.71 KB heapSize=362.55 KB 2023-04-05 09:55:20,170 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,170 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:20,169 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2fc7202cae6c4ad684cd1798b8c9c0bf, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680688515982 2023-04-05 09:55:20,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:20,171 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 375cbdccd33941c38e3d59795e41c845, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680688518142 2023-04-05 09:55:20,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,170 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c46bd4b495284562bb04ccb8c28bef4b, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680688518142 2023-04-05 09:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,171 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:20,171 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:20,171 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d9dda9900d7c4aaa8c5b01fa29799ce7, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:20,171 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 65cc924fee724f0595bbcfc9505e5875, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:20,171 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:20,171 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:20,210 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#B#compaction#329 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:20,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688580198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,226 DEBUG [Thread-2356] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:20,232 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#A#compaction#328 average throughput is 0.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:20,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688580210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688580215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688580216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688580217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,253 DEBUG [Thread-2348] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8217 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:20,270 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,296 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,317 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053315cec3aa854356ad12d590e555fbad to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053315cec3aa854356ad12d590e555fbad 2023-04-05 09:55:20,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,337 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45eb6585[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,344 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688580334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,347 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,351 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688580346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,351 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,355 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,358 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,358 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,358 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688580347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,458 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,462 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,462 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,466 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,466 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,466 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,466 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,468 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b045468[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,471 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,471 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,471 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,476 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,476 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,481 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,485 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,485 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,485 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,485 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688580547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688580552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688580562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,585 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,588 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,589 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,589 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,589 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,591 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,591 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,598 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6e9e03[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,602 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,609 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,611 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,613 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,613 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:20,666 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/5919ba5b310a4cfd9c2afb1c07b9eec0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5919ba5b310a4cfd9c2afb1c07b9eec0 2023-04-05 09:55:20,670 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/B of fb59cf4f6165326988fc5be4f32ae6fb into 5919ba5b310a4cfd9c2afb1c07b9eec0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:20,670 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:20,670 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/B, priority=8, startTime=1680688520158; duration=0sec 2023-04-05 09:55:20,670 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:20,670 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 09:55:20,672 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 93144 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 09:55:20,673 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/C is initiating minor compaction (all files) 2023-04-05 09:55:20,673 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/C in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,673 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=91.0 K 2023-04-05 09:55:20,673 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 41177b76f20440ffa6b628994ae06a1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680688509854 2023-04-05 09:55:20,673 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 547f8cdf59624f41b68b9afc22013d2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680688509896 2023-04-05 09:55:20,674 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d3a06b9b1d2947cc8b1b0fe5ae5a9bb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680688512023 2023-04-05 09:55:20,674 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 974145d2285a47a38308ec0b6c8ac102, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680688514112 2023-04-05 09:55:20,674 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 32dd5c7f02764667941b137644bc56cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680688515265 2023-04-05 09:55:20,674 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 35a03abb430e439fb9fdafae233a7a79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680688515982 2023-04-05 09:55:20,675 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 99662e5282d54f378a6c91c9d04b3fb7, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680688518142 2023-04-05 09:55:20,675 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 09b2afbf182f4674be6eb0e1d2fbad99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:20,688 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#C#compaction#330 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:20,713 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,716 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,720 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,720 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,721 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,721 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,722 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74564a00[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,724 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,724 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,725 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,725 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,730 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,736 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,737 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,738 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,738 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,738 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,738 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/2b1292be370146c58ff645f45b212e11 2023-04-05 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,744 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:20,744 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,744 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,844 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,847 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,848 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,848 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,849 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,853 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0fce42[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,856 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,859 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,859 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688580859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,866 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,867 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,867 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,867 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688580869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688580881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,967 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:20,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:20,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:20,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:20,970 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:20,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:20,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:20,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,971 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,972 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:20,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:20,974 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,974 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:20,974 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:20,974 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,975 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6321607f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:20,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:20,977 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,978 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:20,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:20,981 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:20,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:20,983 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:20,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:20,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:20,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:20,983 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:20,984 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,984 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:20,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,984 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,985 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:20,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:20,988 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:20,988 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:20,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:20,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:20,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:20,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:20,989 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:20,989 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:20,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:21,089 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:21,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:21,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:21,092 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:21,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:21,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:21,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,093 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,093 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,094 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,094 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:21,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:21,096 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,096 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:21,096 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:21,096 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:21,096 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:21,096 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/6f9e2810b50d495ab0b95b6d79a056a2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/6f9e2810b50d495ab0b95b6d79a056a2 2023-04-05 09:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,098 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db1749e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:21,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:21,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,102 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:21,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:21,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,107 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/C of fb59cf4f6165326988fc5be4f32ae6fb into 6f9e2810b50d495ab0b95b6d79a056a2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:21,107 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:21,107 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/C, priority=8, startTime=1680688520159; duration=0sec 2023-04-05 09:55:21,107 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:21,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:21,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:21,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:21,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:21,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:21,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:21,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:21,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,119 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:21,119 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:21,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:21,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/6452e96a75454bb4967b1876c381644c 2023-04-05 09:55:21,219 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:21,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:21,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:21,222 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:21,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:21,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:21,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,224 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:21,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:21,225 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:21,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:21,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:21,227 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b713562[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:21,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:21,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:21,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:21,235 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:21,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:21,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:21,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:21,235 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:21,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,237 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,243 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:21,243 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,244 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:21,244 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:21,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,344 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:21,347 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:21,347 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,347 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:21,347 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:21,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,349 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:21,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:21,351 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,351 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:21,351 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:21,351 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f988390[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:21,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:21,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,356 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:21,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:21,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,357 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:21,361 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:21,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:21,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:21,361 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:21,361 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:21,362 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,362 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,363 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,364 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,366 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,366 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,367 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:21,367 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:21,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:21,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688581367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:21,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688581374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688581390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,467 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:21,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:21,470 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:21,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:21,476 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d23e84[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:21,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:21,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:21,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:21,481 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:21,483 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:21,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:21,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:21,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:21,483 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:21,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:21,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,487 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,493 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,494 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:21,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:21,493 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:21,494 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:21,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:21,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:21,495 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:21,495 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:21,534 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/d9fca1ea4d664f23a3d4accd36d6e7fd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9fca1ea4d664f23a3d4accd36d6e7fd 2023-04-05 09:55:21,537 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/A of fb59cf4f6165326988fc5be4f32ae6fb into d9fca1ea4d664f23a3d4accd36d6e7fd(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 09:55:21,538 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:21,538 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/A, priority=8, startTime=1680688520156; duration=1sec 2023-04-05 09:55:21,538 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:21,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/1e7373cdd4284594bbac9f596f9e8e18 2023-04-05 09:55:21,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/2b1292be370146c58ff645f45b212e11 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11 2023-04-05 09:55:21,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11, entries=250, sequenceid=374, filesize=39.2 K 2023-04-05 09:55:21,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/6452e96a75454bb4967b1876c381644c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c 2023-04-05 09:55:21,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c, entries=150, sequenceid=374, filesize=11.9 K 2023-04-05 09:55:21,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/1e7373cdd4284594bbac9f596f9e8e18 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18 2023-04-05 09:55:21,595 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:21,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18, entries=150, sequenceid=374, filesize=11.9 K 2023-04-05 09:55:21,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:21,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~362.50 KB/371200, currentSize=53.67 KB/54960 for fb59cf4f6165326988fc5be4f32ae6fb in 1426ms, sequenceid=374, compaction requested=false 2023-04-05 09:55:21,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:21,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:21,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:21,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:21,613 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:21,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:21,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:21,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,615 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:21,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:21,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:21,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:21,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:21,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:21,617 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:21,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:21,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:21,620 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:21,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:21,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:21,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:21,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:21,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:21,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:22,054 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052db61a1422864091947209b07b0f7bc1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052db61a1422864091947209b07b0f7bc1 2023-04-05 09:55:22,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:22,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688522404, max timestamp=1680688522416Num uniques 859; 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688522404, max timestamp=1680688522416Num uniques 859; 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688522404, max timestamp=1680688522416Num uniques 859; 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:22,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:22,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688582444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688582444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688582450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,459 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=388, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/f78359e4b91f464f932235b6fa197787 2023-04-05 09:55:22,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688582557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688582557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688582558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688582759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688582765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:22,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688582766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:22,869 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/91098ad750444089bd8ef401ea9f10fb 2023-04-05 09:55:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688583065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688583070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688583070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,278 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/9dc8225de8174970add0b8e0a5eb9885 2023-04-05 09:55:23,282 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/f78359e4b91f464f932235b6fa197787 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787 2023-04-05 09:55:23,286 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787, entries=150, sequenceid=388, filesize=25.4 K 2023-04-05 09:55:23,287 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/91098ad750444089bd8ef401ea9f10fb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb 2023-04-05 09:55:23,290 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb, entries=150, sequenceid=388, filesize=11.9 K 2023-04-05 09:55:23,290 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/9dc8225de8174970add0b8e0a5eb9885 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885 2023-04-05 09:55:23,293 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885, entries=150, sequenceid=388, filesize=11.9 K 2023-04-05 09:55:23,294 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fb59cf4f6165326988fc5be4f32ae6fb in 1674ms, sequenceid=388, compaction requested=true 2023-04-05 09:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:23,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3121a564[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:23,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:23,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:23,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:23,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:23,307 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:23,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:23,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:23,307 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:23,307 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:23,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:23,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:23,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:23,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:23,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:23,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:23,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:23,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:23,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:23,313 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:23,313 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:23,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:23,413 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:23,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:23,416 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:23,416 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:23,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:23,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:23,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:23,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:23,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:23,419 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,419 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:23,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:23,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:23,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:23,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:23,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:23,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:23,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:23,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:23,420 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:23,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:23,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:23,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:23,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688583590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688583592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688583598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688583699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688583699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688583706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,841 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c80087e41455415686e642c3f4c59540 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c80087e41455415686e642c3f4c59540 2023-04-05 09:55:23,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688583904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688583904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:23,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:23,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688583911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688584206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688584206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688584215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,250 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=422, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/eb084fe1b15d400d9e25e878603cbb92 2023-04-05 09:55:24,659 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/2750165c948e4bde899a54bd5ba0f899 2023-04-05 09:55:24,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688584716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688584717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:24,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688584723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,071 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/8699ed8c80894b28ab9d2f8904f040db 2023-04-05 09:55:25,075 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/eb084fe1b15d400d9e25e878603cbb92 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92 2023-04-05 09:55:25,078 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92, entries=150, sequenceid=422, filesize=25.4 K 2023-04-05 09:55:25,079 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/2750165c948e4bde899a54bd5ba0f899 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899 2023-04-05 09:55:25,083 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899, entries=150, sequenceid=422, filesize=11.9 K 2023-04-05 09:55:25,084 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/8699ed8c80894b28ab9d2f8904f040db as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db 2023-04-05 09:55:25,087 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db, entries=150, sequenceid=422, filesize=11.9 K 2023-04-05 09:55:25,088 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for fb59cf4f6165326988fc5be4f32ae6fb in 1668ms, sequenceid=422, compaction requested=true 2023-04-05 09:55:25,088 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:25,088 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,092 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:25,092 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:25,094 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d94c07[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:25,096 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:25,096 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,097 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,097 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:25,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,098 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:25,099 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:25,102 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:25,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:25,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:25,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:25,102 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:25,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:25,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:25,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:25,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:25,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:25,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:25,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:25,109 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,109 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:25,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:25,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:25,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:25,209 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:25,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:25,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:25,212 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:25,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:25,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:25,214 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:25,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:25,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:25,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:25,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:25,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:25,216 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:25,216 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:25,217 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:25,639 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052581c60e4a9b4ae48ebf636ecd5b36ca to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052581c60e4a9b4ae48ebf636ecd5b36ca 2023-04-05 09:55:25,656 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/cf69f81f1e504b66befe370a75ec6f23 2023-04-05 09:55:25,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:25,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:25,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688525788, max timestamp=1680688525813Num uniques 859; 2023-04-05 09:55:25,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688525788, max timestamp=1680688525813Num uniques 859; 2023-04-05 09:55:25,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:25,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:25,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688525788, max timestamp=1680688525813Num uniques 859; 2023-04-05 09:55:25,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688585846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688585855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688585858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688585959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688585964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:25,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:25,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688585964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,070 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/291bfd9632c34521a360123b45c1e02c 2023-04-05 09:55:26,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688586163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688586170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688586170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688586469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688586477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688586478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,486 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/b25e13e8536a4af395fe289346d62337 2023-04-05 09:55:26,490 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/cf69f81f1e504b66befe370a75ec6f23 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23 2023-04-05 09:55:26,493 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23, entries=100, sequenceid=433, filesize=18.6 K 2023-04-05 09:55:26,494 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/291bfd9632c34521a360123b45c1e02c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c 2023-04-05 09:55:26,497 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c, entries=100, sequenceid=433, filesize=9.5 K 2023-04-05 09:55:26,497 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/b25e13e8536a4af395fe289346d62337 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337 2023-04-05 09:55:26,500 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337, entries=100, sequenceid=433, filesize=9.5 K 2023-04-05 09:55:26,501 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fb59cf4f6165326988fc5be4f32ae6fb in 1285ms, sequenceid=433, compaction requested=true 2023-04-05 09:55:26,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:26,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:26,504 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:26,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:26,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,504 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a6d9ef[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:26,505 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:26,505 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,506 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:26,506 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:26,507 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,507 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:26,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:26,508 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:26,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:26,510 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:26,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:26,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:26,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:26,510 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:26,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:26,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:26,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:26,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:26,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:26,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:26,519 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:26,519 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:26,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:26,520 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:26,620 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:26,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:26,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:26,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:26,632 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:26,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:26,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:26,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:26,633 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,633 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:26,634 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:26,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:26,634 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:26,636 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:26,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:26,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:26,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:26,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:26,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:26,643 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 09:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:26,644 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:26,644 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:26,644 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:26,686 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d30bf04a1e254af999092ac9a55884bd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d30bf04a1e254af999092ac9a55884bd 2023-04-05 09:55:26,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:26,980 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:27,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688586997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688587002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688587006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,098 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/79262d6a78364d92acb69a801018377f 2023-04-05 09:55:27,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688587107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,113 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/0ff555d6e10b4c83af902ab4948b53bd 2023-04-05 09:55:27,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688587117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,126 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/b08196c576ae441585b300bed918888d 2023-04-05 09:55:27,130 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/79262d6a78364d92acb69a801018377f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f 2023-04-05 09:55:27,133 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f, entries=150, sequenceid=467, filesize=25.4 K 2023-04-05 09:55:27,134 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/0ff555d6e10b4c83af902ab4948b53bd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd 2023-04-05 09:55:27,137 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 09:55:27,138 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/b08196c576ae441585b300bed918888d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d 2023-04-05 09:55:27,141 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 09:55:27,142 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for fb59cf4f6165326988fc5be4f32ae6fb in 499ms, sequenceid=467, compaction requested=true 2023-04-05 09:55:27,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:27,142 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:27,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e46ec61[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:27,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:27,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,148 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:27,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:27,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:27,155 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:27,155 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:27,155 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:27,155 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:27,155 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:27,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,163 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:27,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:27,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:27,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:27,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:27,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:27,169 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:27,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:27,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:27,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:27,169 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:27,169 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:27,269 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:27,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:27,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:27,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:27,273 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:27,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:27,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:27,274 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:27,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:27,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:27,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:27,276 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:27,276 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:27,276 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:27,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:27,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:27,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688527351, max timestamp=1680688527372Num uniques 859; 2023-04-05 09:55:27,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:27,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688527351, max timestamp=1680688527372Num uniques 859; 2023-04-05 09:55:27,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:27,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688527351, max timestamp=1680688527372Num uniques 859; 2023-04-05 09:55:27,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:27,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688587414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688587420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688587421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688587525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688587530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688587532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,688 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405119487d1575a4132a89a7f71f497c6b3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405119487d1575a4132a89a7f71f497c6b3 2023-04-05 09:55:27,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688587729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688587738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:27,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688587739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688588043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688588043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688588045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,094 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=478, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/79a8ee40b7624b0ba42dcdb197a7c2c0 2023-04-05 09:55:28,510 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/27b73a5982f445afb86cfa990cb5cf42 2023-04-05 09:55:28,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688588552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688588554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688588555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,919 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/cad4d344b57c4c6fadafa6e6e83e369f 2023-04-05 09:55:28,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/79a8ee40b7624b0ba42dcdb197a7c2c0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0 2023-04-05 09:55:28,926 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0, entries=150, sequenceid=478, filesize=25.4 K 2023-04-05 09:55:28,926 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/27b73a5982f445afb86cfa990cb5cf42 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42 2023-04-05 09:55:28,930 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42, entries=150, sequenceid=478, filesize=11.9 K 2023-04-05 09:55:28,930 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/cad4d344b57c4c6fadafa6e6e83e369f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f 2023-04-05 09:55:28,934 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f, entries=150, sequenceid=478, filesize=11.9 K 2023-04-05 09:55:28,934 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fb59cf4f6165326988fc5be4f32ae6fb in 1658ms, sequenceid=478, compaction requested=true 2023-04-05 09:55:28,934 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:28,934 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,937 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f07648[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:28,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:28,939 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:28,940 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,940 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:28,943 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:28,945 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:28,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:28,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:28,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:28,945 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:28,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:28,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:28,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:28,954 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:28,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:28,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:28,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:28,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:28,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:28,955 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:28,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:28,955 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:29,055 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:29,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:29,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:29,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:29,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:29,058 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:29,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:29,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:29,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:29,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:29,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:29,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:29,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:29,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:29,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:29,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:29,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. started... 2023-04-05 09:55:29,062 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:29,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:29,474 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c130ed13e7e7405dabb34b3080f7ce84 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c130ed13e7e7405dabb34b3080f7ce84 2023-04-05 09:55:29,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:29,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. as already flushing 2023-04-05 09:55:29,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688589598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688589599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688589602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688589706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688589710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688589710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,847 DEBUG [Thread-2361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:62753 2023-04-05 09:55:29,847 DEBUG [Thread-2361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:29,847 DEBUG [Thread-2363] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:62753 2023-04-05 09:55:29,847 DEBUG [Thread-2363] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:29,848 DEBUG [Thread-2359] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:62753 2023-04-05 09:55:29,848 DEBUG [Thread-2359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:29,858 DEBUG [Thread-2365] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:62753 2023-04-05 09:55:29,858 DEBUG [Thread-2365] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:29,858 DEBUG [Thread-2367] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:62753 2023-04-05 09:55:29,859 DEBUG [Thread-2367] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:29,879 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=513, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/9170a4906e5c44218fa25da8fe2beea0 2023-04-05 09:55:29,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688589916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:29,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688589917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:29,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688589917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:30,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1680688590217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41356 deadline: 1680688590218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41422 deadline: 1680688590220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:30,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41432 deadline: 1680688590287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,288 DEBUG [Thread-2356] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18257 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:30,289 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/99110756b8774cacb99276f78d74f332 2023-04-05 09:55:30,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41390 deadline: 1680688590340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,342 DEBUG [Thread-2348] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18314 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:30,697 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/eb41c898cee64d5ab9ad8959b15a1e86 2023-04-05 09:55:30,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/9170a4906e5c44218fa25da8fe2beea0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0 2023-04-05 09:55:30,703 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0, entries=150, sequenceid=513, filesize=25.4 K 2023-04-05 09:55:30,703 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/99110756b8774cacb99276f78d74f332 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332 2023-04-05 09:55:30,706 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332, entries=150, sequenceid=513, filesize=11.9 K 2023-04-05 09:55:30,707 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/eb41c898cee64d5ab9ad8959b15a1e86 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86 2023-04-05 09:55:30,709 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86, entries=150, sequenceid=513, filesize=11.9 K 2023-04-05 09:55:30,710 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for fb59cf4f6165326988fc5be4f32ae6fb in 1648ms, sequenceid=513, compaction requested=true 2023-04-05 09:55:30,710 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:30,710 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b10b965[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:30,713 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:30,713 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:30,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,714 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:30,715 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:30,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:30,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:30,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:30,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:30,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:30,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:30,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:30,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:30,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:30,721 DEBUG [Thread-2350] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:62753 2023-04-05 09:55:30,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:30,721 DEBUG [Thread-2350] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:30,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:30,723 DEBUG [Thread-2352] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:62753 2023-04-05 09:55:30,723 DEBUG [Thread-2352] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:30,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:30,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:30,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:30,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:30,726 DEBUG [Thread-2354] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:62753 2023-04-05 09:55:30,726 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:30,726 DEBUG [Thread-2354] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:30,726 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:30,726 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:30,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:30,826 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:31,133 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055b6e9ca4dee44570bbabb98dcb592d82 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055b6e9ca4dee44570bbabb98dcb592d82 2023-04-05 09:55:31,538 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=525, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/286db153f612442e9e0b88ca9eb2cb77 2023-04-05 09:55:31,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/bd47a6e93cc74886a54749c5971f3ceb 2023-04-05 09:55:32,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/0f75a424866544d98c441910fee34f89 2023-04-05 09:55:32,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/286db153f612442e9e0b88ca9eb2cb77 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77 2023-04-05 09:55:32,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77, entries=150, sequenceid=525, filesize=25.4 K 2023-04-05 09:55:32,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/bd47a6e93cc74886a54749c5971f3ceb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb 2023-04-05 09:55:32,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb, entries=150, sequenceid=525, filesize=11.9 K 2023-04-05 09:55:32,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/0f75a424866544d98c441910fee34f89 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89 2023-04-05 09:55:32,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89, entries=150, sequenceid=525, filesize=11.9 K 2023-04-05 09:55:32,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for fb59cf4f6165326988fc5be4f32ae6fb in 1648ms, sequenceid=525, compaction requested=true 2023-04-05 09:55:32,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:32,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:32,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:32,369 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-05 09:55:32,369 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-05 09:55:32,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:32,372 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 241574 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-05 09:55:32,372 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107354 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-05 09:55:32,372 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/A is initiating minor compaction (all files) 2023-04-05 09:55:32,372 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/B is initiating minor compaction (all files) 2023-04-05 09:55:32,372 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/A in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:32,372 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/B in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:32,373 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9fca1ea4d664f23a3d4accd36d6e7fd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=235.9 K 2023-04-05 09:55:32,373 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5919ba5b310a4cfd9c2afb1c07b9eec0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=104.8 K 2023-04-05 09:55:32,373 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d9fca1ea4d664f23a3d4accd36d6e7fd, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:32,373 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5919ba5b310a4cfd9c2afb1c07b9eec0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:32,373 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2b1292be370146c58ff645f45b212e11, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680688520140 2023-04-05 09:55:32,373 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6452e96a75454bb4967b1876c381644c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680688520140 2023-04-05 09:55:32,373 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f78359e4b91f464f932235b6fa197787, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680688520196 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 91098ad750444089bd8ef401ea9f10fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680688520196 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting eb084fe1b15d400d9e25e878603cbb92, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680688522436 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2750165c948e4bde899a54bd5ba0f899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680688522436 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting cf69f81f1e504b66befe370a75ec6f23, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680688523589 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 291bfd9632c34521a360123b45c1e02c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680688523589 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 79262d6a78364d92acb69a801018377f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680688525834 2023-04-05 09:55:32,374 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0ff555d6e10b4c83af902ab4948b53bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680688525834 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 79a8ee40b7624b0ba42dcdb197a7c2c0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680688526996 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 27b73a5982f445afb86cfa990cb5cf42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680688526996 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9170a4906e5c44218fa25da8fe2beea0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680688527413 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 99110756b8774cacb99276f78d74f332, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680688527413 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 286db153f612442e9e0b88ca9eb2cb77, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680688529597 2023-04-05 09:55:32,375 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting bd47a6e93cc74886a54749c5971f3ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680688529597 2023-04-05 09:55:32,387 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#B#compaction#355 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:32,391 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#A#compaction#354 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:32,794 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/5cffa0d81fed4828b161d2e41b9c1f45 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5cffa0d81fed4828b161d2e41b9c1f45 2023-04-05 09:55:32,798 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/B of fb59cf4f6165326988fc5be4f32ae6fb into 5cffa0d81fed4828b161d2e41b9c1f45(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:32,798 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:32,798 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/B, priority=7, startTime=1680688532369; duration=0sec 2023-04-05 09:55:32,798 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:32,798 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-05 09:55:32,800 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107354 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-05 09:55:32,800 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): fb59cf4f6165326988fc5be4f32ae6fb/C is initiating minor compaction (all files) 2023-04-05 09:55:32,800 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb59cf4f6165326988fc5be4f32ae6fb/C in TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:32,800 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/6f9e2810b50d495ab0b95b6d79a056a2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp, totalSize=104.8 K 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6f9e2810b50d495ab0b95b6d79a056a2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680688518915 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1e7373cdd4284594bbac9f596f9e8e18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680688520140 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9dc8225de8174970add0b8e0a5eb9885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680688520196 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8699ed8c80894b28ab9d2f8904f040db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680688522436 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b25e13e8536a4af395fe289346d62337, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680688523589 2023-04-05 09:55:32,801 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b08196c576ae441585b300bed918888d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680688525834 2023-04-05 09:55:32,802 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cad4d344b57c4c6fadafa6e6e83e369f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680688526996 2023-04-05 09:55:32,802 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting eb41c898cee64d5ab9ad8959b15a1e86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680688527413 2023-04-05 09:55:32,802 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0f75a424866544d98c441910fee34f89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680688529597 2023-04-05 09:55:32,811 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): fb59cf4f6165326988fc5be4f32ae6fb#C#compaction#356 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-05 09:55:33,218 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/94ab44a113af4a689a9e4f3bbc7c70c6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/94ab44a113af4a689a9e4f3bbc7c70c6 2023-04-05 09:55:33,221 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/C of fb59cf4f6165326988fc5be4f32ae6fb into 94ab44a113af4a689a9e4f3bbc7c70c6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:33,221 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:33,221 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/C, priority=7, startTime=1680688532369; duration=0sec 2023-04-05 09:55:33,221 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:33,606 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/ebc236f1f9a14d35b38842870f4ac21c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ebc236f1f9a14d35b38842870f4ac21c 2023-04-05 09:55:33,610 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in fb59cf4f6165326988fc5be4f32ae6fb/A of fb59cf4f6165326988fc5be4f32ae6fb into ebc236f1f9a14d35b38842870f4ac21c(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 09:55:33,610 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:33,610 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb., storeName=fb59cf4f6165326988fc5be4f32ae6fb/A, priority=7, startTime=1680688532369; duration=1sec 2023-04-05 09:55:33,610 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:38,585 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9fca1ea4d664f23a3d4accd36d6e7fd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77] to archive 2023-04-05 09:55:38,586 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:38,588 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/b2dde124e3e64f37821cc9c9205ab48c 2023-04-05 09:55:38,589 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/e64e965070b741debf7d5920eb730f4f 2023-04-05 09:55:38,590 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ff461c3275ec4f80a3613abd69150131 2023-04-05 09:55:38,590 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/67dfcc12c8a348cc9c674ccc43cb8671 2023-04-05 09:55:38,591 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f28887de0ad64a619e198346393b49e0 2023-04-05 09:55:38,592 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2fc7202cae6c4ad684cd1798b8c9c0bf 2023-04-05 09:55:38,593 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/375cbdccd33941c38e3d59795e41c845 2023-04-05 09:55:38,594 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9dda9900d7c4aaa8c5b01fa29799ce7 2023-04-05 09:55:38,594 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9fca1ea4d664f23a3d4accd36d6e7fd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/d9fca1ea4d664f23a3d4accd36d6e7fd 2023-04-05 09:55:38,595 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/2b1292be370146c58ff645f45b212e11 2023-04-05 09:55:38,596 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/f78359e4b91f464f932235b6fa197787 2023-04-05 09:55:38,597 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/eb084fe1b15d400d9e25e878603cbb92 2023-04-05 09:55:38,597 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/cf69f81f1e504b66befe370a75ec6f23 2023-04-05 09:55:38,598 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79262d6a78364d92acb69a801018377f 2023-04-05 09:55:38,599 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/79a8ee40b7624b0ba42dcdb197a7c2c0 2023-04-05 09:55:38,600 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/9170a4906e5c44218fa25da8fe2beea0 2023-04-05 09:55:38,600 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/286db153f612442e9e0b88ca9eb2cb77 2023-04-05 09:55:38,610 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5919ba5b310a4cfd9c2afb1c07b9eec0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb] to archive 2023-04-05 09:55:38,611 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:38,612 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/1575355354904c0ba47036a43783b088 2023-04-05 09:55:38,613 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/58212befc97c40f794a4d4333052f472 2023-04-05 09:55:38,613 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/9cd97f08bca94ad0b2f0a96cb847cc2c 2023-04-05 09:55:38,614 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/16c89e67337d483383707517f352b5b8 2023-04-05 09:55:38,615 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/532ef50fab45442a9fe8fae850370d73 2023-04-05 09:55:38,616 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/b2f0046b3c1b4b81b4671879a2859f8a 2023-04-05 09:55:38,616 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/c46bd4b495284562bb04ccb8c28bef4b 2023-04-05 09:55:38,617 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5919ba5b310a4cfd9c2afb1c07b9eec0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5919ba5b310a4cfd9c2afb1c07b9eec0 2023-04-05 09:55:38,618 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/65cc924fee724f0595bbcfc9505e5875 2023-04-05 09:55:38,619 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/6452e96a75454bb4967b1876c381644c 2023-04-05 09:55:38,620 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/91098ad750444089bd8ef401ea9f10fb 2023-04-05 09:55:38,621 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/2750165c948e4bde899a54bd5ba0f899 2023-04-05 09:55:38,622 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/291bfd9632c34521a360123b45c1e02c 2023-04-05 09:55:38,623 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/0ff555d6e10b4c83af902ab4948b53bd 2023-04-05 09:55:38,624 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/27b73a5982f445afb86cfa990cb5cf42 2023-04-05 09:55:38,625 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/99110756b8774cacb99276f78d74f332 2023-04-05 09:55:38,626 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/bd47a6e93cc74886a54749c5971f3ceb 2023-04-05 09:55:38,629 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/6f9e2810b50d495ab0b95b6d79a056a2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89] to archive 2023-04-05 09:55:38,629 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:55:38,630 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/41177b76f20440ffa6b628994ae06a1c 2023-04-05 09:55:38,631 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/547f8cdf59624f41b68b9afc22013d2f 2023-04-05 09:55:38,632 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/d3a06b9b1d2947cc8b1b0fe5ae5a9bb0 2023-04-05 09:55:38,633 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/974145d2285a47a38308ec0b6c8ac102 2023-04-05 09:55:38,634 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/32dd5c7f02764667941b137644bc56cd 2023-04-05 09:55:38,635 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/35a03abb430e439fb9fdafae233a7a79 2023-04-05 09:55:38,636 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/99662e5282d54f378a6c91c9d04b3fb7 2023-04-05 09:55:38,637 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/6f9e2810b50d495ab0b95b6d79a056a2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/6f9e2810b50d495ab0b95b6d79a056a2 2023-04-05 09:55:38,638 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/09b2afbf182f4674be6eb0e1d2fbad99 2023-04-05 09:55:38,638 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/1e7373cdd4284594bbac9f596f9e8e18 2023-04-05 09:55:38,639 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/9dc8225de8174970add0b8e0a5eb9885 2023-04-05 09:55:38,640 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/8699ed8c80894b28ab9d2f8904f040db 2023-04-05 09:55:38,641 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b25e13e8536a4af395fe289346d62337 2023-04-05 09:55:38,642 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/b08196c576ae441585b300bed918888d 2023-04-05 09:55:38,643 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/cad4d344b57c4c6fadafa6e6e83e369f 2023-04-05 09:55:38,643 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/eb41c898cee64d5ab9ad8959b15a1e86 2023-04-05 09:55:38,644 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/0f75a424866544d98c441910fee34f89 2023-04-05 09:55:40,335 DEBUG [Thread-2356] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:62753 2023-04-05 09:55:40,335 DEBUG [Thread-2356] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:40,442 DEBUG [Thread-2348] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:62753 2023-04-05 09:55:40,442 DEBUG [Thread-2348] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:40,442 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1660 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 4980 rows 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1653 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 4958 rows 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1647 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 4938 rows 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1666 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 4994 rows 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 1666 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 4992 rows 2023-04-05 09:55:40,443 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:55:40,443 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:62753 2023-04-05 09:55:40,443 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:55:40,445 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:55:40,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:55:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:40,449 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688540449"}]},"ts":"1680688540449"} 2023-04-05 09:55:40,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:40,450 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:55:40,452 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:55:40,453 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, UNASSIGN}] 2023-04-05 09:55:40,454 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, UNASSIGN 2023-04-05 09:55:40,454 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:40,454 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688540454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688540454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688540454"}]},"ts":"1680688540454"} 2023-04-05 09:55:40,455 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:40,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:40,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb59cf4f6165326988fc5be4f32ae6fb, disabling compactions & flushes 2023-04-05 09:55:40,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. after waiting 0 ms 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:40,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fb59cf4f6165326988fc5be4f32ae6fb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=A 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=B 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb59cf4f6165326988fc5be4f32ae6fb, store=C 2023-04-05 09:55:40,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:40,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:41,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040522455a0dcae347f79a9f7ccb59f8c721 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040522455a0dcae347f79a9f7ccb59f8c721 2023-04-05 09:55:41,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:41,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=535, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/ecb7602bf3934aabae3e0e2f96f14391 2023-04-05 09:55:41,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:41,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/fa5bf9cfe11f4c329519ba79ed2a253f 2023-04-05 09:55:42,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/a6178d2cdc584d528cc159c7279de6ff 2023-04-05 09:55:42,244 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/A/ecb7602bf3934aabae3e0e2f96f14391 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ecb7602bf3934aabae3e0e2f96f14391 2023-04-05 09:55:42,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ecb7602bf3934aabae3e0e2f96f14391, entries=150, sequenceid=535, filesize=25.4 K 2023-04-05 09:55:42,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/B/fa5bf9cfe11f4c329519ba79ed2a253f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/fa5bf9cfe11f4c329519ba79ed2a253f 2023-04-05 09:55:42,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/fa5bf9cfe11f4c329519ba79ed2a253f, entries=150, sequenceid=535, filesize=11.9 K 2023-04-05 09:55:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/.tmp/C/a6178d2cdc584d528cc159c7279de6ff as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/a6178d2cdc584d528cc159c7279de6ff 2023-04-05 09:55:42,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/a6178d2cdc584d528cc159c7279de6ff, entries=150, sequenceid=535, filesize=11.9 K 2023-04-05 09:55:42,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for fb59cf4f6165326988fc5be4f32ae6fb in 1646ms, sequenceid=535, compaction requested=false 2023-04-05 09:55:42,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits/538.seqid, newMaxSeqId=538, maxSeqId=4 2023-04-05 09:55:42,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb. 2023-04-05 09:55:42,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb59cf4f6165326988fc5be4f32ae6fb: 2023-04-05 09:55:42,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:42,263 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=fb59cf4f6165326988fc5be4f32ae6fb, regionState=CLOSED 2023-04-05 09:55:42,263 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688542263"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688542263"}]},"ts":"1680688542263"} 2023-04-05 09:55:42,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-05 09:55:42,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure fb59cf4f6165326988fc5be4f32ae6fb, server=jenkins-hbase4.apache.org,46827,1680688416991 in 1.8090 sec 2023-04-05 09:55:42,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-05 09:55:42,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb59cf4f6165326988fc5be4f32ae6fb, UNASSIGN in 1.8120 sec 2023-04-05 09:55:42,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688542266"}]},"ts":"1680688542266"} 2023-04-05 09:55:42,267 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:55:42,268 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:55:42,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8240 sec 2023-04-05 09:55:42,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 09:55:42,553 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-05 09:55:42,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:55:42,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,555 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 09:55:42,556 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,559 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:42,561 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits] 2023-04-05 09:55:42,563 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ebc236f1f9a14d35b38842870f4ac21c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ebc236f1f9a14d35b38842870f4ac21c 2023-04-05 09:55:42,564 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ecb7602bf3934aabae3e0e2f96f14391 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/A/ecb7602bf3934aabae3e0e2f96f14391 2023-04-05 09:55:42,566 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5cffa0d81fed4828b161d2e41b9c1f45 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/5cffa0d81fed4828b161d2e41b9c1f45 2023-04-05 09:55:42,567 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/fa5bf9cfe11f4c329519ba79ed2a253f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/B/fa5bf9cfe11f4c329519ba79ed2a253f 2023-04-05 09:55:42,568 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/94ab44a113af4a689a9e4f3bbc7c70c6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/94ab44a113af4a689a9e4f3bbc7c70c6 2023-04-05 09:55:42,570 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/a6178d2cdc584d528cc159c7279de6ff to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/C/a6178d2cdc584d528cc159c7279de6ff 2023-04-05 09:55:42,572 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits/538.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb/recovered.edits/538.seqid 2023-04-05 09:55:42,573 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/fb59cf4f6165326988fc5be4f32ae6fb 2023-04-05 09:55:42,573 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:55:42,573 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:55:42,574 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 09:55:42,577 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405119487d1575a4132a89a7f71f497c6b3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405119487d1575a4132a89a7f71f497c6b3 2023-04-05 09:55:42,578 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405127a82a541cf4c8c98e568f6992052f9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405127a82a541cf4c8c98e568f6992052f9 2023-04-05 09:55:42,579 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040522455a0dcae347f79a9f7ccb59f8c721 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040522455a0dcae347f79a9f7ccb59f8c721 2023-04-05 09:55:42,580 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052581c60e4a9b4ae48ebf636ecd5b36ca to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052581c60e4a9b4ae48ebf636ecd5b36ca 2023-04-05 09:55:42,581 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052db61a1422864091947209b07b0f7bc1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052db61a1422864091947209b07b0f7bc1 2023-04-05 09:55:42,583 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052f8bfa6df8714e25889425d135f2e3e0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052f8bfa6df8714e25889425d135f2e3e0 2023-04-05 09:55:42,584 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405307cb127e43c4837846c3a26dbe9168b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405307cb127e43c4837846c3a26dbe9168b 2023-04-05 09:55:42,585 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040532bb2a8573f34ff48048756d45a1fd9e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040532bb2a8573f34ff48048756d45a1fd9e 2023-04-05 09:55:42,586 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053315cec3aa854356ad12d590e555fbad to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053315cec3aa854356ad12d590e555fbad 2023-04-05 09:55:42,587 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055b6e9ca4dee44570bbabb98dcb592d82 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055b6e9ca4dee44570bbabb98dcb592d82 2023-04-05 09:55:42,588 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040585dfcd784f1f4c9d809c52e5f5f43032 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040585dfcd784f1f4c9d809c52e5f5f43032 2023-04-05 09:55:42,589 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aa9f52923805405b820accdca92b9fab to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aa9f52923805405b820accdca92b9fab 2023-04-05 09:55:42,590 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b34472e6c3024081acd4ab7c2d24af5f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b34472e6c3024081acd4ab7c2d24af5f 2023-04-05 09:55:42,591 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c130ed13e7e7405dabb34b3080f7ce84 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c130ed13e7e7405dabb34b3080f7ce84 2023-04-05 09:55:42,592 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c80087e41455415686e642c3f4c59540 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c80087e41455415686e642c3f4c59540 2023-04-05 09:55:42,593 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d30bf04a1e254af999092ac9a55884bd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d30bf04a1e254af999092ac9a55884bd 2023-04-05 09:55:42,594 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fbf0f04b99424c5da9ce9c6b58536f51 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fbf0f04b99424c5da9ce9c6b58536f51 2023-04-05 09:55:42,595 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:55:42,596 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,599 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:55:42,603 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:55:42,604 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,604 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:55:42,604 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688542604"}]},"ts":"9223372036854775807"} 2023-04-05 09:55:42,606 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:55:42,606 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fb59cf4f6165326988fc5be4f32ae6fb, NAME => 'TestAcidGuarantees,,1680688507770.fb59cf4f6165326988fc5be4f32ae6fb.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:55:42,606 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:55:42,606 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688542606"}]},"ts":"9223372036854775807"} 2023-04-05 09:55:42,607 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:55:42,608 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,609 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 55 msec 2023-04-05 09:55:42,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 09:55:42,657 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-05 09:55:42,668 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 228), OpenFileDescriptor=511 (was 552), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=323 (was 405), ProcessCount=169 (was 169), AvailableMemoryMB=896 (was 1023) 2023-04-05 09:55:42,679 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=511, MaxFileDescriptor=60000, SystemLoadAverage=323, ProcessCount=169, AvailableMemoryMB=896 2023-04-05 09:55:42,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:55:42,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:55:42,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:55:42,683 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:55:42,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-05 09:55:42,683 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:55:42,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:42,684 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:42,685 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc empty. 2023-04-05 09:55:42,685 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:42,685 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:55:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:42,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:43,092 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:55:43,093 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 62afd6a1e0fb2ca72b38ec7d7142b5cc, NAME => 'TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:55:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 62afd6a1e0fb2ca72b38ec7d7142b5cc, disabling compactions & flushes 2023-04-05 09:55:43,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. after waiting 0 ms 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:43,501 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:55:43,501 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688543501"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688543501"}]},"ts":"1680688543501"} 2023-04-05 09:55:43,502 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:55:43,503 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:55:43,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688543503"}]},"ts":"1680688543503"} 2023-04-05 09:55:43,504 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:55:43,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, ASSIGN}] 2023-04-05 09:55:43,513 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, ASSIGN 2023-04-05 09:55:43,514 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:55:43,665 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=62afd6a1e0fb2ca72b38ec7d7142b5cc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:43,665 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688543665"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688543665"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688543665"}]},"ts":"1680688543665"} 2023-04-05 09:55:43,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:55:43,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:43,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 62afd6a1e0fb2ca72b38ec7d7142b5cc, NAME => 'TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:55:43,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:55:43,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,821 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,822 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A 2023-04-05 09:55:43,822 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A 2023-04-05 09:55:43,823 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:43,823 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62afd6a1e0fb2ca72b38ec7d7142b5cc columnFamilyName A 2023-04-05 09:55:43,823 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(310): Store=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:43,823 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,824 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B 2023-04-05 09:55:43,824 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B 2023-04-05 09:55:43,824 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:43,824 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62afd6a1e0fb2ca72b38ec7d7142b5cc columnFamilyName B 2023-04-05 09:55:43,825 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(310): Store=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:43,825 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,826 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C 2023-04-05 09:55:43,826 DEBUG [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C 2023-04-05 09:55:43,826 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:55:43,826 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62afd6a1e0fb2ca72b38ec7d7142b5cc columnFamilyName C 2023-04-05 09:55:43,826 INFO [StoreOpener-62afd6a1e0fb2ca72b38ec7d7142b5cc-1] regionserver.HStore(310): Store=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:55:43,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:55:43,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:55:43,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 62afd6a1e0fb2ca72b38ec7d7142b5cc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66275507, jitterRate=-0.012417986989021301}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:55:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:43,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., pid=49, masterSystemTime=1680688543817 2023-04-05 09:55:43,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:43,833 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=62afd6a1e0fb2ca72b38ec7d7142b5cc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:43,834 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688543833"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688543833"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688543833"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688543833"}]},"ts":"1680688543833"} 2023-04-05 09:55:43,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-05 09:55:43,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 in 168 msec 2023-04-05 09:55:43,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-05 09:55:43,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, ASSIGN in 323 msec 2023-04-05 09:55:43,837 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:55:43,837 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688543837"}]},"ts":"1680688543837"} 2023-04-05 09:55:43,838 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:55:43,839 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:55:43,840 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-05 09:55:44,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 09:55:44,788 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-05 09:55:44,788 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x02df63a4 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,793 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@501c5a88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,794 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,796 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,797 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:55:44,798 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:55:44,799 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,803 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,803 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,807 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,808 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,811 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,811 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,814 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,814 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,818 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,818 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,824 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,824 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,827 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,827 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,832 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,833 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,839 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,839 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:55:44,843 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:55:44,858 DEBUG [hconnection-0x6e1d00f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,858 DEBUG [hconnection-0x3d907a5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,858 DEBUG [hconnection-0x4182c38f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,858 DEBUG [hconnection-0x300a9c3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,859 DEBUG [hconnection-0x13e15042-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,859 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,859 DEBUG [hconnection-0x2f7b5370-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,859 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,861 DEBUG [hconnection-0x1677fdf8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,861 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,864 DEBUG [hconnection-0x65ada4b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:44,865 DEBUG [hconnection-0x548daaac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,865 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,865 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,867 DEBUG [hconnection-0xb7dd009-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:55:44,868 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,868 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,868 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,869 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,871 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:55:44,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:44,877 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:44,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,878 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:44,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:44,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:44,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:44,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:44,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:44,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:44,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:44,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:44,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:44,883 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:44,883 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:44,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:44,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:44,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:44,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688604901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:44,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688604902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688544896, max timestamp=1680688544900Num uniques 859; 2023-04-05 09:55:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688604901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:44,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688604902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688544896, max timestamp=1680688544901Num uniques 859; 2023-04-05 09:55:44,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 09:55:44,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680688544896, max timestamp=1680688544901Num uniques 859; 2023-04-05 09:55:44,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 09:55:44,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:44,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688604920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:44,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15277849[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:44,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:44,927 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:44,929 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,929 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:44,931 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:44,933 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:44,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:44,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:44,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:44,933 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:44,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,934 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:44,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:44,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:44,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:44,938 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:44,939 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:44,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:44,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:44,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,939 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:44,939 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:44,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:44,939 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:44,939 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:44,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:44,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688605003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688605004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688605004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688605005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688605021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,039 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,046 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,046 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,048 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,052 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,052 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,052 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,052 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,055 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,055 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9bbb68[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,062 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,062 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,063 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,065 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,065 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,065 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,067 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,067 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,067 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,075 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,075 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,075 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,076 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,076 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,176 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,180 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,180 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,184 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,185 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,188 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,188 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,188 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,188 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,189 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,191 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d48de85[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,191 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,192 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,192 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,194 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,194 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,194 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,195 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,197 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,197 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,198 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,198 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,198 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,199 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,202 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,204 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,204 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,204 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,204 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688605209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688605210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688605211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688605211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688605228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,305 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,309 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,312 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,312 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,312 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,312 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@535d97d8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,316 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,317 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,322 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,323 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,324 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1bea23dc71184e339f4bf1308b6d354f 2023-04-05 09:55:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,327 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,330 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:45,331 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,331 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/d0abf23e016d482f99f761d2403a541e 2023-04-05 09:55:45,431 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,437 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d9658ca[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,446 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,446 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,447 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,448 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,451 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,452 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,456 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,464 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,464 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,464 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688605516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688605516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688605520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688605520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688605535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,564 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,567 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,568 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,570 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,570 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,570 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,570 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,574 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,575 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c87fceb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,577 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,581 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,581 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,582 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,583 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,589 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,589 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,589 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,589 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,589 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,689 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,693 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,694 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,697 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,697 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,697 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:45,697 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:45,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7760a308[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:45,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:45,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:45,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:45,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,703 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:45,706 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:45,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:45,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:45,706 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:45,706 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:45,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:45,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:45,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,709 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:45,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:45,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:45,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:45,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:45,712 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,712 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:45,712 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:45,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:45,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/019b47a5bbae4dabbdb470fd7fc34f99 2023-04-05 09:55:45,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1bea23dc71184e339f4bf1308b6d354f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f 2023-04-05 09:55:45,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f, entries=100, sequenceid=12, filesize=9.3 K 2023-04-05 09:55:45,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/d0abf23e016d482f99f761d2403a541e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e 2023-04-05 09:55:45,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e, entries=100, sequenceid=12, filesize=9.3 K 2023-04-05 09:55:45,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/019b47a5bbae4dabbdb470fd7fc34f99 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99 2023-04-05 09:55:45,812 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:45,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:45,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99, entries=100, sequenceid=12, filesize=9.3 K 2023-04-05 09:55:45,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 932ms, sequenceid=12, compaction requested=false 2023-04-05 09:55:45,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:45,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:45,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:45,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:45,818 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:45,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:45,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:45,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,820 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:45,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:45,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:45,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:45,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:45,822 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:45,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:45,823 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:45,823 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:45,842 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/a95a4b52edc44ef3a50916f4c936b3a4 2023-04-05 09:55:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:46,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688606031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688606032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688606033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688606036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688606041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688606137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688606139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688606139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688606140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,276 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/6e0f9da122c0444ba57c8173841232a7 2023-04-05 09:55:46,286 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/5c82305d38c04762bb72361fba57915f 2023-04-05 09:55:46,289 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/a95a4b52edc44ef3a50916f4c936b3a4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4 2023-04-05 09:55:46,294 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 09:55:46,296 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/6e0f9da122c0444ba57c8173841232a7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7 2023-04-05 09:55:46,300 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 09:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/5c82305d38c04762bb72361fba57915f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f 2023-04-05 09:55:46,306 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 09:55:46,307 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=53.67 KB/54960 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 485ms, sequenceid=48, compaction requested=false 2023-04-05 09:55:46,307 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:46,307 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,310 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bc096b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,312 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,312 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,318 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,318 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,318 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,318 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,318 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,322 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,324 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,328 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,328 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,329 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:46,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:55:46,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:46,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:46,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:46,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:46,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:46,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:46,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688606383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688546381, max timestamp=1680688546388Num uniques 859; 2023-04-05 09:55:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:46,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688546381, max timestamp=1680688546388Num uniques 859; 2023-04-05 09:55:46,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:46,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688546381, max timestamp=1680688546388Num uniques 859; 2023-04-05 09:55:46,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:46,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688606414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688606419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688606420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,429 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:46,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:46,432 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:46,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:46,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:46,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,432 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,434 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:46,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a348be[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,443 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,443 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,444 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,451 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,451 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,451 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,451 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,451 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:46,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,458 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,458 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688606490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688606524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688606525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688606526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,558 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:46,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:46,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:46,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:46,561 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:46,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:46,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:46,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,563 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:46,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:46,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:46,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,565 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14509c05[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,570 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,570 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,572 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,574 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,574 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,575 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,580 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:46,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,582 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,582 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,682 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:46,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:46,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:46,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:46,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:46,685 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:46,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:46,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:46,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,686 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:46,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:46,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:46,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:46,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:46,689 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,689 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:46,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,690 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66da0564[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,693 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,694 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,694 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,696 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688606697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,702 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,702 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,702 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,703 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,704 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,704 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,711 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,711 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688606729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688606736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:46,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688606736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd 2023-04-05 09:55:46,811 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:46,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:46,814 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:46,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:46,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:46,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,815 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:46,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:46,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:46,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:46,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,817 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:46,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:46,819 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,819 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:46,819 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,819 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,821 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e05bf85[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,829 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,829 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,838 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,838 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,838 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,938 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:46,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:46,942 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:46,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:46,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:46,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,943 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,944 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:46,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:46,945 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,945 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:46,945 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:46,945 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:46,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:46,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537ef843[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:46,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:46,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,951 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:46,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:46,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:46,956 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:46,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:46,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:46,956 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:46,956 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:46,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:46,958 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,960 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:46,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:46,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:46,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:46,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:46,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:46,962 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:46,962 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:46,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:46,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688607002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688607036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688607046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688607047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688607047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,063 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:47,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:47,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:47,066 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:47,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:47,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:47,071 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,071 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,073 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,073 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,074 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,074 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:47,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:47,076 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,076 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:47,076 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:47,076 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:47,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,079 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435ef69d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:47,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:47,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,083 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:47,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:47,084 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,084 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:47,084 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:47,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:47,087 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:47,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:47,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:47,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:47,087 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:47,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,089 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,090 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:47,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:47,103 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,104 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:47,104 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:47,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:47,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/35043553249f40919142fe900ac50c6c 2023-04-05 09:55:47,204 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:47,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:47,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:47,208 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:47,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:47,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:47,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,208 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,209 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,209 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:47,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:47,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:47,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:47,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:47,211 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,211 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:47,211 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:47,211 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,214 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:47,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75993ab7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:47,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:47,216 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,218 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:47,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:47,218 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:47,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:47,219 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,219 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:47,219 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,219 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:47,222 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:47,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:47,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:47,222 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:47,222 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:47,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:47,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:47,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,232 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:47,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,232 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:47,232 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:47,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,332 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:47,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:47,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:47,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:47,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:47,336 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:47,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:47,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:47,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,337 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,337 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:47,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:47,339 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,339 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:47,339 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:47,339 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:47,340 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16834a16[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:47,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:47,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,343 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:47,343 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,344 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:47,349 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:47,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:47,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:47,351 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:47,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:47,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:47,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:47,351 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:47,352 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,352 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,352 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,355 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:47,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,358 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:47,358 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:47,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,357 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:47,458 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:47,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:47,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:47,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:47,476 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:47,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:47,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:47,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,478 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:47,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:47,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:47,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:47,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:47,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:47,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329b1465[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:47,485 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:47,485 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:47,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:47,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:47,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:47,487 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:47,489 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:47,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:47,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:47,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:47,489 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:47,490 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:47,491 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,491 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:47,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,492 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,493 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,496 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:47,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,496 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:47,497 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:47,496 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:47,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688607511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688607543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688607554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:47,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688607554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/c9397e082ac24e49ac5e3874c16e4df6 2023-04-05 09:55:47,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd 2023-04-05 09:55:47,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd, entries=200, sequenceid=60, filesize=13.9 K 2023-04-05 09:55:47,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/35043553249f40919142fe900ac50c6c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c 2023-04-05 09:55:47,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 09:55:47,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/c9397e082ac24e49ac5e3874c16e4df6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6 2023-04-05 09:55:47,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 09:55:47,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 1250ms, sequenceid=60, compaction requested=true 2023-04-05 09:55:47,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:47,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:47,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:47,593 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:47,593 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:47,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:47,594 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:47,594 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:47,594 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,594 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=34.8 K 2023-04-05 09:55:47,594 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:47,595 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1bea23dc71184e339f4bf1308b6d354f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680688544875 2023-04-05 09:55:47,595 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=32.5 K 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a95a4b52edc44ef3a50916f4c936b3a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688544916 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d0abf23e016d482f99f761d2403a541e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680688544875 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c2a0ecddca5a4a65b4d51cfebb7e7cdd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546030 2023-04-05 09:55:47,595 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6e0f9da122c0444ba57c8173841232a7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688544916 2023-04-05 09:55:47,596 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 35043553249f40919142fe900ac50c6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546031 2023-04-05 09:55:47,597 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:47,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:47,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:47,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:47,612 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:47,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:47,612 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:47,613 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:47,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,615 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:47,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:47,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:47,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:47,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:47,621 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:47,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:47,627 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,627 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:47,627 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:47,628 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:47,636 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/516d62310f3942c4bd36adb81895723a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/516d62310f3942c4bd36adb81895723a 2023-04-05 09:55:47,639 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 516d62310f3942c4bd36adb81895723a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:47,639 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:47,639 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=13, startTime=1680688547593; duration=0sec 2023-04-05 09:55:47,639 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:47,639 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:47,640 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:47,640 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:47,641 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:47,641 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=32.5 K 2023-04-05 09:55:47,641 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 019b47a5bbae4dabbdb470fd7fc34f99, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680688544875 2023-04-05 09:55:47,641 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5c82305d38c04762bb72361fba57915f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680688544916 2023-04-05 09:55:47,641 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c9397e082ac24e49ac5e3874c16e4df6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546031 2023-04-05 09:55:47,650 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:47,661 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4a936ad8e5d04ecf82a701ea5bdaa6f6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4a936ad8e5d04ecf82a701ea5bdaa6f6 2023-04-05 09:55:47,664 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 4a936ad8e5d04ecf82a701ea5bdaa6f6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:47,664 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:47,664 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=13, startTime=1680688547593; duration=0sec 2023-04-05 09:55:47,664 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:47,795 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:55:48,036 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/eb2d5bebd87b49549ec0a1f70cf1005c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/eb2d5bebd87b49549ec0a1f70cf1005c 2023-04-05 09:55:48,040 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into eb2d5bebd87b49549ec0a1f70cf1005c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:48,040 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:48,040 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=13, startTime=1680688547593; duration=0sec 2023-04-05 09:55:48,040 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:48,056 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/8f6464d4f2bd45c89cfe5d3008641f44 2023-04-05 09:55:48,067 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/997040c7c28e4b59aacff90cb45db939 2023-04-05 09:55:48,075 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/1fee5ee2dcc64722b209f8ac41c78333 2023-04-05 09:55:48,078 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/8f6464d4f2bd45c89cfe5d3008641f44 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44 2023-04-05 09:55:48,081 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44, entries=150, sequenceid=93, filesize=11.6 K 2023-04-05 09:55:48,082 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/997040c7c28e4b59aacff90cb45db939 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939 2023-04-05 09:55:48,085 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939, entries=150, sequenceid=93, filesize=11.6 K 2023-04-05 09:55:48,085 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/1fee5ee2dcc64722b209f8ac41c78333 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333 2023-04-05 09:55:48,088 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333, entries=150, sequenceid=93, filesize=11.6 K 2023-04-05 09:55:48,089 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 461ms, sequenceid=93, compaction requested=false 2023-04-05 09:55:48,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:48,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,090 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645f6b5[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,094 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,094 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,094 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,097 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,099 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,099 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,100 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,100 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,100 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,106 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,106 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,206 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,212 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,212 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,217 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,220 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,220 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,220 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,221 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,221 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,221 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,221 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,222 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:48,222 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,223 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a352848[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,226 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,226 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,227 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,230 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,230 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,236 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,237 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,237 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,237 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,337 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,341 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,342 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,343 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,343 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,346 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ea16e2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,350 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,351 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,352 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,354 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,354 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,354 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,356 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,356 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,356 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,361 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:48,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,361 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,361 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,361 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,461 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,464 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,467 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,470 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,470 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,470 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:48,470 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,472 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb80466[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,475 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,475 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,476 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,479 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,479 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,485 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,485 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,485 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,486 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,486 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:48,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:48,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:48,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/3153aa8210f44681af2191d22c45b483 2023-04-05 09:55:48,586 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680688548560, max timestamp=1680688548587Num uniques 859; 2023-04-05 09:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680688548560, max timestamp=1680688548587Num uniques 859; 2023-04-05 09:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680688548560, max timestamp=1680688548587Num uniques 859; 2023-04-05 09:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:48,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,590 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,590 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,590 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,590 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,591 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,591 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,598 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e1b11a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,612 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,617 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,620 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,621 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688608618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688608619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,623 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,625 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688608620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688608620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,629 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,629 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688608722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688608723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,729 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688608727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,732 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,733 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,734 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688608730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:48,736 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,737 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6685a086[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,740 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,741 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,741 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,746 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,746 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,749 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,751 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:48,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,752 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,752 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,752 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,852 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,855 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:48,858 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:48,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:48,859 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:48,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef7b2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:48,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:48,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:48,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,864 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:48,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:48,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:48,867 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:48,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:48,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:48,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:48,868 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:48,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:48,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:48,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,879 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:48,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:48,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:48,880 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,880 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:48,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:48,880 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:48,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688608928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688608929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688608932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:48,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688608937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/e7e2f61335a84b55b44f8a06dc57935a 2023-04-05 09:55:48,980 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:48,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:48,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:48,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/dcd292dc78124249abb93b6346b7faa7 2023-04-05 09:55:48,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:48,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:48,992 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:48,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:48,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:48,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/3153aa8210f44681af2191d22c45b483 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483 2023-04-05 09:55:48,994 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:48,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:48,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:48,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:48,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:48,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:48,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:48,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:48,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483, entries=150, sequenceid=109, filesize=11.6 K 2023-04-05 09:55:48,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/e7e2f61335a84b55b44f8a06dc57935a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a 2023-04-05 09:55:49,005 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,006 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,006 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,006 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,006 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,009 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb948e7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,012 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,020 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,020 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,024 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,027 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,027 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,027 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,027 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a, entries=150, sequenceid=109, filesize=11.6 K 2023-04-05 09:55:49,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/dcd292dc78124249abb93b6346b7faa7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7 2023-04-05 09:55:49,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7, entries=150, sequenceid=109, filesize=11.6 K 2023-04-05 09:55:49,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 489ms, sequenceid=109, compaction requested=true 2023-04-05 09:55:49,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:49,036 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:49,036 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:49,037 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:49,037 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:49,037 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:49,037 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:49,038 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,038 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,038 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/eb2d5bebd87b49549ec0a1f70cf1005c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=34.9 K 2023-04-05 09:55:49,038 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/516d62310f3942c4bd36adb81895723a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=34.9 K 2023-04-05 09:55:49,038 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting eb2d5bebd87b49549ec0a1f70cf1005c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546031 2023-04-05 09:55:49,038 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 516d62310f3942c4bd36adb81895723a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546031 2023-04-05 09:55:49,038 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8f6464d4f2bd45c89cfe5d3008641f44, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680688546412 2023-04-05 09:55:49,039 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 997040c7c28e4b59aacff90cb45db939, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680688546412 2023-04-05 09:55:49,039 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 3153aa8210f44681af2191d22c45b483, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:49,039 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e7e2f61335a84b55b44f8a06dc57935a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:49,053 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:49,054 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#379 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:49,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 09:55:49,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:49,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:49,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:49,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:49,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:49,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:49,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/88ac8e3ec883435197f6fcdd4be07cfc 2023-04-05 09:55:49,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688609104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,127 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,144 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a8530b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,162 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,163 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,166 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,166 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,166 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,172 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,172 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,174 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,174 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,174 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,175 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,182 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,184 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,184 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688609211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688609232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688609237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688609238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688609243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,284 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,287 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,287 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,288 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,290 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,290 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,290 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,290 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,292 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f625e88[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,302 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,307 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,308 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,314 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,314 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,414 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,417 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,419 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,419 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,419 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,419 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,420 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688609417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,424 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5896afb1[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,425 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,428 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,431 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,440 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,440 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,441 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:49,469 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/107e96cbd7d641819c246e9752cf6408 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/107e96cbd7d641819c246e9752cf6408 2023-04-05 09:55:49,473 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 107e96cbd7d641819c246e9752cf6408(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:49,473 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:49,473 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=13, startTime=1680688549036; duration=0sec 2023-04-05 09:55:49,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:49,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:49,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:49,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:49,475 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,475 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4a936ad8e5d04ecf82a701ea5bdaa6f6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=34.9 K 2023-04-05 09:55:49,475 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4a936ad8e5d04ecf82a701ea5bdaa6f6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680688546031 2023-04-05 09:55:49,475 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1fee5ee2dcc64722b209f8ac41c78333, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680688546412 2023-04-05 09:55:49,476 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting dcd292dc78124249abb93b6346b7faa7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:49,480 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#382 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:49,481 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/f56a1019e05e47508b28c99307caf0bc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/f56a1019e05e47508b28c99307caf0bc 2023-04-05 09:55:49,486 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into f56a1019e05e47508b28c99307caf0bc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:49,486 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:49,487 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=13, startTime=1680688549036; duration=0sec 2023-04-05 09:55:49,487 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:49,490 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/b084d0de4357463ab919a18472a72ded as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/b084d0de4357463ab919a18472a72ded 2023-04-05 09:55:49,496 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into b084d0de4357463ab919a18472a72ded(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:49,496 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:49,496 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=13, startTime=1680688549036; duration=0sec 2023-04-05 09:55:49,496 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:49,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/24c393b4e0774a4599f5212be3ae85c5 2023-04-05 09:55:49,541 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,544 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,545 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,545 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,546 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,549 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,549 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,549 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,549 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,555 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3e3a34[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,559 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,563 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,563 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,564 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,567 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:49,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,568 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,568 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,668 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,674 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,677 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,677 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,677 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,677 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,679 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,679 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,679 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,684 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,685 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a810f3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,688 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,695 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,695 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,695 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,700 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,702 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,705 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,706 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,706 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,705 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:49,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688609722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688609740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688609744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688609745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:49,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688609753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,806 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,809 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,811 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,812 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,813 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15839d70[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,821 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,821 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,823 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,831 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,831 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,831 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:49,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,932 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:49,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:49,934 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:49,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:49,934 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:49,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,936 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,937 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:49,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:49,938 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,938 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:49,938 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:49,938 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:49,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78054356[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:49,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:49,941 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,942 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:49,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:49,945 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:49,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:49,947 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:49,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:49,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:49,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:49,947 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:49,949 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,949 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,949 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:49,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,949 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:49,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/ba9b142aaa5c42d0a54bee2a7b6d179e 2023-04-05 09:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/88ac8e3ec883435197f6fcdd4be07cfc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc 2023-04-05 09:55:49,954 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:49,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:49,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:49,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:49,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:49,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:49,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:49,955 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:49,955 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:49,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc, entries=200, sequenceid=146, filesize=14.1 K 2023-04-05 09:55:49,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/24c393b4e0774a4599f5212be3ae85c5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5 2023-04-05 09:55:49,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5, entries=150, sequenceid=146, filesize=11.8 K 2023-04-05 09:55:49,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/ba9b142aaa5c42d0a54bee2a7b6d179e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e 2023-04-05 09:55:49,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e, entries=150, sequenceid=146, filesize=11.8 K 2023-04-05 09:55:49,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 900ms, sequenceid=146, compaction requested=false 2023-04-05 09:55:49,967 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,056 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:50,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:50,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:50,059 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:50,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:50,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:50,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,059 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,060 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,060 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:50,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:50,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:50,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:50,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:50,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:50,062 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:50,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:50,062 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 09:55:50,062 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:50,063 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,063 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:50,063 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,063 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:50,063 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,070 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/8c0000414a5b457b82cec33cf9e53267 2023-04-05 09:55:50,078 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/0051f91164d148bfacb948ee0025983c 2023-04-05 09:55:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:50,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688550283, max timestamp=1680688550321Num uniques 859; 2023-04-05 09:55:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688550283, max timestamp=1680688550321Num uniques 859; 2023-04-05 09:55:50,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:50,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688550283, max timestamp=1680688550321Num uniques 859; 2023-04-05 09:55:50,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:50,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688550362, max timestamp=1680688550479Num uniques 1018; 2023-04-05 09:55:50,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:50,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688550362, max timestamp=1680688550479Num uniques 1018; 2023-04-05 09:55:50,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680688550362, max timestamp=1680688550479Num uniques 1018; 2023-04-05 09:55:50,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:50,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 09:55:50,487 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/6fd72310226e44ceb68bf7f720d1d430 2023-04-05 09:55:50,491 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/8c0000414a5b457b82cec33cf9e53267 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267 2023-04-05 09:55:50,495 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 09:55:50,496 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/0051f91164d148bfacb948ee0025983c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c 2023-04-05 09:55:50,500 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 09:55:50,500 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/6fd72310226e44ceb68bf7f720d1d430 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430 2023-04-05 09:55:50,504 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 09:55:50,504 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=46.96 KB/48090 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 442ms, sequenceid=158, compaction requested=true 2023-04-05 09:55:50,505 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,505 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:50,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:50,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-05 09:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:50,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:50,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79529f8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:50,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:50,510 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,511 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:50,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:50,511 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:50,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:50,516 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:50,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:50,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:50,516 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:50,516 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:50,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:50,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:50,524 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:50,524 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,524 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,524 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,524 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/01f1d58d65394cbd86c5323b43b67029 2023-04-05 09:55:50,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/1aa4806a3bd14e10a506f64038a91a8b 2023-04-05 09:55:50,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9c56baee977a47de9a6dbc86908b7bbf 2023-04-05 09:55:50,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/01f1d58d65394cbd86c5323b43b67029 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029 2023-04-05 09:55:50,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029, entries=150, sequenceid=200, filesize=11.8 K 2023-04-05 09:55:50,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/1aa4806a3bd14e10a506f64038a91a8b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b 2023-04-05 09:55:50,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b, entries=150, sequenceid=200, filesize=11.8 K 2023-04-05 09:55:50,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9c56baee977a47de9a6dbc86908b7bbf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf 2023-04-05 09:55:50,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf, entries=150, sequenceid=200, filesize=11.8 K 2023-04-05 09:55:50,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=67.09 KB/68700 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 92ms, sequenceid=200, compaction requested=true 2023-04-05 09:55:50,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 09:55:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:50,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:50,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,598 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:50,599 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:50,600 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50590 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:50,600 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:50,600 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,600 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/f56a1019e05e47508b28c99307caf0bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=49.4 K 2023-04-05 09:55:50,601 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:50,601 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f56a1019e05e47508b28c99307caf0bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:50,601 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:50,601 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,601 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/b084d0de4357463ab919a18472a72ded, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=47.1 K 2023-04-05 09:55:50,603 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 88ac8e3ec883435197f6fcdd4be07cfc, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680688548608 2023-04-05 09:55:50,603 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b084d0de4357463ab919a18472a72ded, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:50,603 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8c0000414a5b457b82cec33cf9e53267, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680688549093 2023-04-05 09:55:50,604 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ba9b142aaa5c42d0a54bee2a7b6d179e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680688548618 2023-04-05 09:55:50,604 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 01f1d58d65394cbd86c5323b43b67029, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:50,604 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6fd72310226e44ceb68bf7f720d1d430, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680688549093 2023-04-05 09:55:50,605 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9c56baee977a47de9a6dbc86908b7bbf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:50,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/6d4e719c8e8549cbab46c9b9934e308f 2023-04-05 09:55:50,613 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:50,622 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#392 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:50,624 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:50,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:50,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:50,632 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:50,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:50,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:50,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,632 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,633 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,633 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:50,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:50,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:50,652 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/f3df85725e3c485a96bc29c2cf312b9e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f3df85725e3c485a96bc29c2cf312b9e 2023-04-05 09:55:50,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/600c6102571247c2aa046d2fbcdb7446 2023-04-05 09:55:50,656 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into f3df85725e3c485a96bc29c2cf312b9e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:50,656 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,657 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:50,657 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:50,656 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,657 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=12, startTime=1680688550598; duration=0sec 2023-04-05 09:55:50,657 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:50,657 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:50,657 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:50,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:50,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,659 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:50,659 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/a986652e78fe41b5a63d2a4c1a581cd8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a986652e78fe41b5a63d2a4c1a581cd8 2023-04-05 09:55:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,660 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:50,661 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,661 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,661 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/107e96cbd7d641819c246e9752cf6408, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=47.1 K 2023-04-05 09:55:50,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:50,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd29e71[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:50,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:50,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,663 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 107e96cbd7d641819c246e9752cf6408, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680688548541 2023-04-05 09:55:50,663 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 24c393b4e0774a4599f5212be3ae85c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680688548618 2023-04-05 09:55:50,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,664 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0051f91164d148bfacb948ee0025983c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680688549093 2023-04-05 09:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:50,664 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1aa4806a3bd14e10a506f64038a91a8b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,666 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into a986652e78fe41b5a63d2a4c1a581cd8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:50,666 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,666 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=12, startTime=1680688550597; duration=0sec 2023-04-05 09:55:50,666 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:50,668 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:50,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:50,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:50,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:50,669 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:50,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,673 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:50,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:50,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,674 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,674 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:50,675 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:50,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,675 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,675 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/ff11fbcb85934af68506630884ab8ce6 2023-04-05 09:55:50,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/6d4e719c8e8549cbab46c9b9934e308f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f 2023-04-05 09:55:50,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f, entries=200, sequenceid=214, filesize=14.1 K 2023-04-05 09:55:50,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/600c6102571247c2aa046d2fbcdb7446 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446 2023-04-05 09:55:50,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446, entries=150, sequenceid=214, filesize=11.8 K 2023-04-05 09:55:50,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/ff11fbcb85934af68506630884ab8ce6 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6 2023-04-05 09:55:50,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6, entries=150, sequenceid=214, filesize=11.8 K 2023-04-05 09:55:50,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=87.22 KB/89310 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 96ms, sequenceid=214, compaction requested=false 2023-04-05 09:55:50,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:50,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:50,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:50,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:50,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688610750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688610751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688610755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688610756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688610757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,775 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:50,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:50,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:50,778 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:50,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:50,790 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,790 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:50,790 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:50,790 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,792 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6633f6e6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:50,793 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:50,793 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:50,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:50,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:50,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:50,796 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:50,798 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:50,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:50,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:50,798 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:50,798 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:50,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,803 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,804 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,805 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,807 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:50,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,807 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:50,807 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688610859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,907 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:50,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:50,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:50,910 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:50,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:50,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:50,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,911 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,912 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:50,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:50,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:50,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:50,914 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:50,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:50,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34fe1685[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:50,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:50,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,918 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,920 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:50,922 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:50,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:50,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:50,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:50,922 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:50,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,924 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:50,925 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:50,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:50,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:50,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:50,928 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:50,928 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,928 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:50,929 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,029 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,033 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,040 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,040 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,040 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,040 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,044 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,046 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0a330b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,050 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,051 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,051 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,051 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,052 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,052 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,055 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,055 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,055 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,058 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,061 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,062 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,062 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,062 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,062 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,062 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45140 deadline: 1680688611062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,086 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/39622020f3534a73a0dda8fe9635e88a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/39622020f3534a73a0dda8fe9635e88a 2023-04-05 09:55:51,090 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 39622020f3534a73a0dda8fe9635e88a(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:51,090 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:51,090 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=12, startTime=1680688550597; duration=0sec 2023-04-05 09:55:51,091 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:51,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/c8a1733e7d9a4f4f9144323ee98bc8d8 2023-04-05 09:55:51,162 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,165 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,167 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,168 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,169 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,174 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6150172d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,177 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,177 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,181 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,181 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,181 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,182 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,183 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,187 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,187 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,188 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,188 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,288 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,291 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,291 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,292 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,292 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,293 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,294 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182d4927[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,298 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,303 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,303 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,303 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,304 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,305 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,306 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,309 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,308 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,309 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688611367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,409 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,412 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,412 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,412 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,414 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,415 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,419 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c233e99[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,421 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,422 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,424 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,427 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,427 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,431 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,437 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,437 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/4c4a5685e113442b8209258dc43174b3 2023-04-05 09:55:51,537 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,540 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,540 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,542 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177acc7d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,547 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,548 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,553 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,555 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,555 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,556 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,556 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,557 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,557 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,557 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,559 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,562 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,562 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,562 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,662 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,665 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,668 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,668 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f248c0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,675 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,676 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,677 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,679 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,680 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,680 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,681 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,683 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,686 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,686 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,687 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,687 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,687 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,687 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,787 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,791 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,792 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,793 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,794 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,794 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,794 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,800 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,802 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e44348[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,804 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,804 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,807 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,807 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,807 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,807 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,809 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,810 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,810 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,810 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,813 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,813 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,814 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,816 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,817 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,817 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,816 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,817 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688611872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,917 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:51,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:51,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/83e2ae03b5d24930bc8f05b97e71f1fd 2023-04-05 09:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:51,920 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:51,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:51,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:51,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,921 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,922 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,922 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:51,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:51,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/c8a1733e7d9a4f4f9144323ee98bc8d8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8 2023-04-05 09:55:51,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:51,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:51,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:51,924 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:51,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8, entries=200, sequenceid=233, filesize=14.1 K 2023-04-05 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:51,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/4c4a5685e113442b8209258dc43174b3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3 2023-04-05 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417672bb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:51,929 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:51,929 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,930 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:51,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:51,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3, entries=150, sequenceid=233, filesize=11.8 K 2023-04-05 09:55:51,932 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/83e2ae03b5d24930bc8f05b97e71f1fd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd 2023-04-05 09:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:51,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:51,935 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:51,935 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:51,935 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:51,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd, entries=150, sequenceid=233, filesize=11.8 K 2023-04-05 09:55:51,936 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 1241ms, sequenceid=233, compaction requested=true 2023-04-05 09:55:51,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:51,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:51,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:51,937 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:51,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:51,937 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:51,937 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:51,938 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,938 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,938 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a986652e78fe41b5a63d2a4c1a581cd8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=40.3 K 2023-04-05 09:55:51,938 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/39622020f3534a73a0dda8fe9635e88a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=35.6 K 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a986652e78fe41b5a63d2a4c1a581cd8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 39622020f3534a73a0dda8fe9635e88a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6d4e719c8e8549cbab46c9b9934e308f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680688550520 2023-04-05 09:55:51,938 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 600c6102571247c2aa046d2fbcdb7446, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680688550520 2023-04-05 09:55:51,939 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c8a1733e7d9a4f4f9144323ee98bc8d8, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:51,939 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4c4a5685e113442b8209258dc43174b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:51,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:51,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,940 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:51,940 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,942 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:51,944 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:51,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:51,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:51,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:51,944 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:51,944 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:51,945 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:51,945 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:51,945 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:51,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:51,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:51,946 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#400 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:51,952 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/73803ee68807437397a34800b280a29f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73803ee68807437397a34800b280a29f 2023-04-05 09:55:51,954 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/9ab771643cee4f25b79a85b13ccf8472 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/9ab771643cee4f25b79a85b13ccf8472 2023-04-05 09:55:51,956 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 73803ee68807437397a34800b280a29f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:51,956 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:51,956 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=13, startTime=1680688551937; duration=0sec 2023-04-05 09:55:51,956 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:51,957 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:51,957 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:51,957 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 9ab771643cee4f25b79a85b13ccf8472(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:51,957 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:51,957 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:51,958 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=13, startTime=1680688551937; duration=0sec 2023-04-05 09:55:51,958 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:51,958 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:51,958 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f3df85725e3c485a96bc29c2cf312b9e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=35.6 K 2023-04-05 09:55:51,958 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f3df85725e3c485a96bc29c2cf312b9e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680688550479 2023-04-05 09:55:51,958 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ff11fbcb85934af68506630884ab8ce6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680688550520 2023-04-05 09:55:51,958 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 83e2ae03b5d24930bc8f05b97e71f1fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:51,963 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:51,969 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/a4acb7cd37b5444c90dfab5d1c00ac1a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/a4acb7cd37b5444c90dfab5d1c00ac1a 2023-04-05 09:55:51,972 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into a4acb7cd37b5444c90dfab5d1c00ac1a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:51,972 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:51,972 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=13, startTime=1680688551937; duration=0sec 2023-04-05 09:55:51,972 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:52,045 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:52,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:52,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:52,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:52,047 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:52,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:52,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:52,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,048 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,049 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:52,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:52,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,049 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:52,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:52,051 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:52,051 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,058 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/2a0e70665b2b48129652a69d9893e8fa 2023-04-05 09:55:52,073 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/b148f700255847b0b9312bedf1f36a80 2023-04-05 09:55:52,485 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4d8ebb2f8ed54eb0bd09c388d720d296 2023-04-05 09:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/2a0e70665b2b48129652a69d9893e8fa as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa 2023-04-05 09:55:52,491 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa, entries=150, sequenceid=252, filesize=11.8 K 2023-04-05 09:55:52,492 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/b148f700255847b0b9312bedf1f36a80 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80 2023-04-05 09:55:52,495 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80, entries=150, sequenceid=252, filesize=11.8 K 2023-04-05 09:55:52,496 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4d8ebb2f8ed54eb0bd09c388d720d296 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296 2023-04-05 09:55:52,499 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296, entries=150, sequenceid=252, filesize=11.8 K 2023-04-05 09:55:52,500 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 449ms, sequenceid=252, compaction requested=false 2023-04-05 09:55:52,500 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:52,500 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:52,501 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e57cc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:52,503 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:52,504 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,506 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:52,506 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,506 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:52,507 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:52,511 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:52,511 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:52,511 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:52,511 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:52,511 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:52,512 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:52,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:52,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:52,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:52,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,517 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:52,517 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:52,517 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:52,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:52,617 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:52,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:52,621 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,626 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:52,626 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:52,626 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,626 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:52,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:52,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:52,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:52,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:52,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:52,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:52,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:52,635 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64febb41[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:52,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:52,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,639 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:52,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:52,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:52,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:52,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:52,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:52,645 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:52,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:52,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:52,645 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:52,645 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:52,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,653 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:52,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:52,654 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,655 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,657 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:52,657 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:52,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:52,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:52,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:52,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:52,657 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:52,657 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:52,659 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,758 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:52,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:52,761 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:52,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:52,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:52,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,762 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:52,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:52,763 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,763 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:52,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:52,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:52,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:52,765 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:52,765 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,766 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:52,766 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:52,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:52,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:52,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688552787, max timestamp=1680688552794Num uniques 859; 2023-04-05 09:55:52,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:52,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688552787, max timestamp=1680688552794Num uniques 859; 2023-04-05 09:55:52,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:52,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688552787, max timestamp=1680688552794Num uniques 859; 2023-04-05 09:55:52,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:52,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688612813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688612815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688612816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688612819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688612878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688612919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688612922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688612922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:52,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688612926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688613130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688613130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688613130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688613131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,174 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/cbd1232496844538854da037b2639aaf 2023-04-05 09:55:53,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688613439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688613439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688613440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688613442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,584 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/847b444bd3bf42d4b229d5f867ef6f11 2023-04-05 09:55:53,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688613945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688613949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688613949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688613949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:53,993 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/1fd752e52da049729178d1844e7f213b 2023-04-05 09:55:53,997 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/cbd1232496844538854da037b2639aaf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf 2023-04-05 09:55:54,001 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf, entries=150, sequenceid=261, filesize=11.9 K 2023-04-05 09:55:54,002 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/847b444bd3bf42d4b229d5f867ef6f11 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11 2023-04-05 09:55:54,005 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11, entries=100, sequenceid=261, filesize=9.5 K 2023-04-05 09:55:54,005 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/1fd752e52da049729178d1844e7f213b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b 2023-04-05 09:55:54,009 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b, entries=100, sequenceid=261, filesize=9.5 K 2023-04-05 09:55:54,010 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 1245ms, sequenceid=261, compaction requested=true 2023-04-05 09:55:54,010 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:54,010 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:54,012 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,012 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a65a95[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:54,014 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:54,014 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,016 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:54,016 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,017 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:54,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:54,020 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:54,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:54,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:54,021 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:54,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,024 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,030 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,030 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,030 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,030 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,030 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,031 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,031 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,031 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:54,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:54,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:54,031 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,032 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:54,032 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:54,132 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:54,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:54,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:54,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:54,135 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:54,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:54,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:54,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,137 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:54,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:54,143 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:54,144 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:54,144 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:54,552 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/ceab9ee3d636485d9196aed228da8857 2023-04-05 09:55:54,562 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/26f8240f4d734a03bf484531ff0106c8 2023-04-05 09:55:54,572 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/42298ac2ac6e41c9967101512a1a9101 2023-04-05 09:55:54,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/ceab9ee3d636485d9196aed228da8857 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857 2023-04-05 09:55:54,580 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857, entries=150, sequenceid=298, filesize=11.9 K 2023-04-05 09:55:54,581 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/26f8240f4d734a03bf484531ff0106c8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8 2023-04-05 09:55:54,584 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8, entries=150, sequenceid=298, filesize=11.9 K 2023-04-05 09:55:54,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/42298ac2ac6e41c9967101512a1a9101 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101 2023-04-05 09:55:54,588 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101, entries=150, sequenceid=298, filesize=11.9 K 2023-04-05 09:55:54,588 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 444ms, sequenceid=298, compaction requested=true 2023-04-05 09:55:54,588 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:54,588 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,590 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:54,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30371bd0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:54,599 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:54,599 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,601 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,601 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:54,601 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:54,601 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:54,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:54,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:54,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:54,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:54,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:54,605 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:54,605 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:54,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,607 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,611 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:54,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:54,614 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:54,614 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:54,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:54,714 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:54,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:54,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:54,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:54,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:54,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,720 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,720 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,720 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,720 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:54,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:54,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:54,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:54,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:54,722 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:54,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255ba2d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:54,727 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:54,727 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,728 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:54,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:54,729 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:54,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:54,730 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:54,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,730 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:54,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:54,734 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:54,734 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:54,736 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,736 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,736 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,738 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,742 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,742 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:54,742 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,842 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:54,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:54,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:54,847 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:54,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:54,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:54,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,848 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,849 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:54,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:54,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:54,851 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:54,851 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:54,853 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53603595[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:54,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:54,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:54,858 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,858 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:54,859 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:54,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:54,862 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:54,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:54,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:54,862 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:54,862 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:54,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,863 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:54,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,874 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,875 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:54,875 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:54,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:54,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:54,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:55:54,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:54,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:54,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:54,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:54,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:54,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:54,975 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:54,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:54,978 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:54,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:54,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:54,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,978 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:54,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:54,981 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,981 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:54,981 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:54,981 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:54,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2362af9e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:54,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:54,984 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,986 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:54,986 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:54,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,986 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:54,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:54,989 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:54,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:54,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:54,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:54,989 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:54,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688554980, max timestamp=1680688554990Num uniques 859; 2023-04-05 09:55:54,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:54,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688614983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688554980, max timestamp=1680688554990Num uniques 859; 2023-04-05 09:55:54,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:54,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688554980, max timestamp=1680688554990Num uniques 859; 2023-04-05 09:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:54,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:55:54,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,992 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,992 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,993 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:54,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:54,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:54,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:54,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:54,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:54,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:54,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:54,995 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:54,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:54,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:54,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:54,996 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:54,996 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:54,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688615018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688615019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688615019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688615026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,096 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:55,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:55,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688615093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:55,099 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:55,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:55,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:55,099 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,102 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:55,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:55,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:55,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:55,103 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:55,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b03c9b9[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:55,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:55,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:55,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:55,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:55,111 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:55,111 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:55,111 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:55,111 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:55,111 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:55,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,116 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,118 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:55,119 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,119 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:55,119 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:55,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688615127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688615127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688615127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688615127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,219 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:55,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:55,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:55,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:55,223 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:55,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:55,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:55,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,223 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,224 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,224 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,225 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:55,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:55,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:55,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:55,226 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:55,228 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e89a26[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:55,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:55,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:55,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,234 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:55,236 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:55,236 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:55,236 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:55,236 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:55,236 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:55,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,238 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,239 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:55,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:55,241 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,241 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,242 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,242 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:55,242 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:55,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:55,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688615299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/610bdef38aff4da9a911fdfae3e6c427 2023-04-05 09:55:55,333 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/5f28f15108cc4d38a23ad011e88d2464 2023-04-05 09:55:55,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688615335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,342 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:55,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/f6593224209b44aeb38720cc72efdfcd 2023-04-05 09:55:55,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:55,345 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:55,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:55,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:55,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688615338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,346 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,346 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688615338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688615338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/610bdef38aff4da9a911fdfae3e6c427 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427 2023-04-05 09:55:55,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:55,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:55,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427, entries=150, sequenceid=310, filesize=11.9 K 2023-04-05 09:55:55,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/5f28f15108cc4d38a23ad011e88d2464 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464 2023-04-05 09:55:55,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,354 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f6c3b3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:55,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:55,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464, entries=150, sequenceid=310, filesize=11.9 K 2023-04-05 09:55:55,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/f6593224209b44aeb38720cc72efdfcd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd 2023-04-05 09:55:55,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:55,361 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:55,362 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd, entries=150, sequenceid=310, filesize=11.9 K 2023-04-05 09:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 452ms, sequenceid=310, compaction requested=true 2023-04-05 09:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:55,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:55,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,363 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:55,364 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:55,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:55,365 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:55,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:55,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:55,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:55,365 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:55,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61070 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:55,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:55,365 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,365 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/9ab771643cee4f25b79a85b13ccf8472, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=59.6 K 2023-04-05 09:55:55,365 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58630 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:55,366 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:55,366 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,366 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73803ee68807437397a34800b280a29f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=57.3 K 2023-04-05 09:55:55,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9ab771643cee4f25b79a85b13ccf8472, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:55,366 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 73803ee68807437397a34800b280a29f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:55,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2a0e70665b2b48129652a69d9893e8fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680688550726 2023-04-05 09:55:55,366 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b148f700255847b0b9312bedf1f36a80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680688550726 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cbd1232496844538854da037b2639aaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680688552761 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 847b444bd3bf42d4b229d5f867ef6f11, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680688552761 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ceab9ee3d636485d9196aed228da8857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680688552812 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 26f8240f4d734a03bf484531ff0106c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680688552812 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 610bdef38aff4da9a911fdfae3e6c427, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:55,367 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5f28f15108cc4d38a23ad011e88d2464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:55,368 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,368 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,368 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,368 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,370 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,373 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,373 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,373 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:55,373 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:55,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:55,373 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,374 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:55,374 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:55,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,374 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,377 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:55,379 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:55,387 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/20b697b9d5564cacb7ed5331a8415526 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/20b697b9d5564cacb7ed5331a8415526 2023-04-05 09:55:55,391 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 20b697b9d5564cacb7ed5331a8415526(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:55,391 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:55,391 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=11, startTime=1680688555363; duration=0sec 2023-04-05 09:55:55,391 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:55,391 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:55:55,392 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58630 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:55:55,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:55,393 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,393 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/a4acb7cd37b5444c90dfab5d1c00ac1a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=57.3 K 2023-04-05 09:55:55,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a4acb7cd37b5444c90dfab5d1c00ac1a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680688550624 2023-04-05 09:55:55,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4d8ebb2f8ed54eb0bd09c388d720d296, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680688550726 2023-04-05 09:55:55,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1fd752e52da049729178d1844e7f213b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680688552761 2023-04-05 09:55:55,394 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 42298ac2ac6e41c9967101512a1a9101, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680688552812 2023-04-05 09:55:55,394 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f6593224209b44aeb38720cc72efdfcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:55,401 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:55,474 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:55,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:55,477 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:55,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:55,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:55,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,478 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:55,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:55,480 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:55,486 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/d15b5d7adfc54619b279a73196d90e4a 2023-04-05 09:55:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:55,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:55,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688615649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688615651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688615652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688615652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688615653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688615763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688615763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688615763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688615763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,793 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/a2777c5401a6426c9bfc981f217417d0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a2777c5401a6426c9bfc981f217417d0 2023-04-05 09:55:55,797 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into a2777c5401a6426c9bfc981f217417d0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:55,797 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:55,797 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=11, startTime=1680688555363; duration=0sec 2023-04-05 09:55:55,797 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:55,808 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/615fd19830ef4120aae534f1a70504d9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/615fd19830ef4120aae534f1a70504d9 2023-04-05 09:55:55,812 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 615fd19830ef4120aae534f1a70504d9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:55,812 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:55,812 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=11, startTime=1680688555363; duration=0sec 2023-04-05 09:55:55,812 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:55,898 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/fe4436e7713f4987857ebc0f732de6f4 2023-04-05 09:55:55,909 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/0192155b3be7400a8149d7386ed713fc 2023-04-05 09:55:55,912 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/d15b5d7adfc54619b279a73196d90e4a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a 2023-04-05 09:55:55,915 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a, entries=150, sequenceid=346, filesize=11.9 K 2023-04-05 09:55:55,916 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/fe4436e7713f4987857ebc0f732de6f4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4 2023-04-05 09:55:55,919 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4, entries=150, sequenceid=346, filesize=11.9 K 2023-04-05 09:55:55,919 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/0192155b3be7400a8149d7386ed713fc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc 2023-04-05 09:55:55,922 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc, entries=150, sequenceid=346, filesize=11.9 K 2023-04-05 09:55:55,923 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 443ms, sequenceid=346, compaction requested=false 2023-04-05 09:55:55,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:55,923 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:55,925 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,925 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@205e90ac[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:55,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:55,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:55,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,928 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:55,929 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:55,932 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:55,932 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:55,932 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:55,932 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:55,932 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:55,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,935 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:55,936 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,940 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:55,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:55,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:55,950 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,950 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:55,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:55,950 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:55,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:55,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:55,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:55,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:55,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:55,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:55,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:55,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:56,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,050 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:56,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:56,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:56,054 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:56,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:56,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:56,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,054 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,055 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,055 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,056 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:56,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:56,057 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,058 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:56,058 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:56,058 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:56,059 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1525db1f[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:56,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:56,061 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,063 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:56,063 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,063 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:56,064 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:56,066 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:56,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:56,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:56,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:56,067 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:56,068 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,068 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,068 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,072 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:56,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,072 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,072 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:56,073 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:56,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688616156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,173 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:56,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:56,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:56,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:56,176 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:56,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:56,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:56,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,176 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,178 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,178 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:56,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:56,179 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,179 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:56,179 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:56,181 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@151c4a37[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:56,182 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:56,182 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,183 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:56,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,184 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:56,185 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:56,187 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:56,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:56,187 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:56,188 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:56,188 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:56,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,189 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,189 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,190 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,192 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,192 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,192 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:56,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,193 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,193 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,193 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,293 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:56,296 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:56,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:56,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:56,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,296 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,297 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,297 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:56,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:56,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:56,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:56,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:56,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,302 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d7f87c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:56,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:56,304 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,305 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:56,305 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,305 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:56,306 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:56,308 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:56,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:56,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:56,308 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:56,308 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:56,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,310 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:56,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,314 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,314 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,314 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/cb756e2b42d24549a75106b5e7035c57 2023-04-05 09:55:56,414 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:56,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:56,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/5ffe3a088e0b4254875b7fe352ceaaf7 2023-04-05 09:55:56,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:56,424 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:56,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:56,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:56,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,426 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:56,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:56,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:56,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:56,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:56,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:56,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:56,442 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/15ee156c062d4dc887ba3104c9a8f6aa 2023-04-05 09:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d8d59a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:56,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:56,445 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,446 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:56,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:56,451 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,451 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:56,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/cb756e2b42d24549a75106b5e7035c57 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57 2023-04-05 09:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:56,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:56,455 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:56,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:56,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:56,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:56,455 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:56,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,457 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:56,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57, entries=150, sequenceid=360, filesize=11.9 K 2023-04-05 09:55:56,457 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 09:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:56,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/5ffe3a088e0b4254875b7fe352ceaaf7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7 2023-04-05 09:55:56,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:56,463 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:56,463 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:56,463 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:56,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7, entries=150, sequenceid=360, filesize=11.9 K 2023-04-05 09:55:56,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/15ee156c062d4dc887ba3104c9a8f6aa as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa 2023-04-05 09:55:56,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa, entries=150, sequenceid=360, filesize=11.9 K 2023-04-05 09:55:56,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 506ms, sequenceid=360, compaction requested=true 2023-04-05 09:55:56,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:56,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:56,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:56,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:55:56,484 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:56,484 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:56,484 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:56,485 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,485 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/20b697b9d5564cacb7ed5331a8415526, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=36.3 K 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:56,485 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 20b697b9d5564cacb7ed5331a8415526, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:56,485 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a2777c5401a6426c9bfc981f217417d0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=36.3 K 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d15b5d7adfc54619b279a73196d90e4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680688555017 2023-04-05 09:55:56,485 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting a2777c5401a6426c9bfc981f217417d0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:56,486 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cb756e2b42d24549a75106b5e7035c57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:56,487 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting fe4436e7713f4987857ebc0f732de6f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680688555017 2023-04-05 09:55:56,487 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 5ffe3a088e0b4254875b7fe352ceaaf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:56,501 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:56,503 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:56,555 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/60f15fbe71214421bb16044a09634adf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/60f15fbe71214421bb16044a09634adf 2023-04-05 09:55:56,559 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 60f15fbe71214421bb16044a09634adf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:56,559 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:56,559 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=13, startTime=1680688556484; duration=0sec 2023-04-05 09:55:56,559 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:56,559 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:55:56,560 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:55:56,560 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:56,560 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,560 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/615fd19830ef4120aae534f1a70504d9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=36.3 K 2023-04-05 09:55:56,561 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 615fd19830ef4120aae534f1a70504d9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680688554896 2023-04-05 09:55:56,561 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 0192155b3be7400a8149d7386ed713fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680688555017 2023-04-05 09:55:56,561 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 15ee156c062d4dc887ba3104c9a8f6aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:56,563 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:56,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:56,567 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#425 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:56,567 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:56,567 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:56,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:56,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,568 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,569 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:56,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:56,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,569 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:56,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:56,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:56,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:56,584 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:56,584 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:56,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:56,585 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:56,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:56,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,925 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/c460b88f2e35473dbc7ad6add1b01010 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c460b88f2e35473dbc7ad6add1b01010 2023-04-05 09:55:56,928 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into c460b88f2e35473dbc7ad6add1b01010(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:56,929 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:56,929 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=13, startTime=1680688556483; duration=0sec 2023-04-05 09:55:56,929 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:56,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688616941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688616944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688616945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688616945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:56,994 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/7c5506cff0bc4ced9a24f859bae90eea as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/7c5506cff0bc4ced9a24f859bae90eea 2023-04-05 09:55:56,998 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 7c5506cff0bc4ced9a24f859bae90eea(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:56,998 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:56,998 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=13, startTime=1680688556484; duration=0sec 2023-04-05 09:55:56,998 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:57,000 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/fe4413de39e74580930d17e4ccf80e3b 2023-04-05 09:55:57,017 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/291aa7feb909460fa145c20c1084dd7c 2023-04-05 09:55:57,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688617160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45216 deadline: 1680688617247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45180 deadline: 1680688617248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688617248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688617249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,434 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9a4d1d84560343b39d162d3a351ce055 2023-04-05 09:55:57,438 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/fe4413de39e74580930d17e4ccf80e3b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b 2023-04-05 09:55:57,442 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 09:55:57,443 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/291aa7feb909460fa145c20c1084dd7c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c 2023-04-05 09:55:57,447 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 09:55:57,448 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9a4d1d84560343b39d162d3a351ce055 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055 2023-04-05 09:55:57,451 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 09:55:57,452 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 868ms, sequenceid=381, compaction requested=false 2023-04-05 09:55:57,452 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:57,452 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:57,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a1e179d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:57,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:57,457 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,458 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:57,461 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:57,461 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:57,461 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:57,465 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,466 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,468 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,470 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:57,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,470 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:57,471 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:57,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,571 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:57,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:57,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:57,574 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:57,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,574 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:57,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:57,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,575 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,576 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,576 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,577 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:57,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:57,579 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,579 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:57,580 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:57,580 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,588 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1ad490c302314e6ebe917ccce3c97962 2023-04-05 09:55:57,596 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/609537646fe84038bbbf6baf9b2b16f0 2023-04-05 09:55:57,604 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/bf629865374e4d48a959d0113065a089 2023-04-05 09:55:57,610 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1ad490c302314e6ebe917ccce3c97962 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962 2023-04-05 09:55:57,615 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962, entries=150, sequenceid=395, filesize=11.9 K 2023-04-05 09:55:57,616 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/609537646fe84038bbbf6baf9b2b16f0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0 2023-04-05 09:55:57,620 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0, entries=150, sequenceid=395, filesize=11.9 K 2023-04-05 09:55:57,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/bf629865374e4d48a959d0113065a089 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089 2023-04-05 09:55:57,633 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089, entries=150, sequenceid=395, filesize=11.9 K 2023-04-05 09:55:57,634 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 54ms, sequenceid=395, compaction requested=true 2023-04-05 09:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:57,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da4b31b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:57,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:57,639 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:57,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,642 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:57,643 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:57,647 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:57,647 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:57,647 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:57,647 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:57,647 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:57,656 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,656 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,664 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,668 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,668 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,669 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:57,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:57,669 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,670 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:57,670 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:57,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:57,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:57,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:57,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/de27c67557424755b8e20b040b4561f7 2023-04-05 09:55:57,770 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:57,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:57,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:57,773 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:57,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:57,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:57,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,776 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:57,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/57b2ef6c35c7427682944c06118223be 2023-04-05 09:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:57,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:57,779 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,779 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:57,779 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:57,779 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:57,780 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,780 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:57,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273abacb[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:57,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:57,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:57,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:57,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:57,787 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:57,787 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:57,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/412e11a94411443990427620b093c98e 2023-04-05 09:55:57,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/de27c67557424755b8e20b040b4561f7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7 2023-04-05 09:55:57,795 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,795 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,795 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7, entries=200, sequenceid=405, filesize=14.3 K 2023-04-05 09:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/57b2ef6c35c7427682944c06118223be as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be 2023-04-05 09:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,799 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:57,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:57,799 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,800 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:57,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,800 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:57,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,800 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be, entries=150, sequenceid=405, filesize=11.9 K 2023-04-05 09:55:57,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/412e11a94411443990427620b093c98e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e 2023-04-05 09:55:57,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688557791, max timestamp=1680688557802Num uniques 859; 2023-04-05 09:55:57,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:57,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688557791, max timestamp=1680688557802Num uniques 859; 2023-04-05 09:55:57,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:57,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688557791, max timestamp=1680688557802Num uniques 859; 2023-04-05 09:55:57,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:57,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e, entries=150, sequenceid=405, filesize=11.9 K 2023-04-05 09:55:57,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=33.54 KB/34350 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 45ms, sequenceid=405, compaction requested=true 2023-04-05 09:55:57,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:57,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:57,807 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:57,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:57,807 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:57,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:57,808 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:57,808 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:57,808 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:55:57,808 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:55:57,808 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,808 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,808 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c460b88f2e35473dbc7ad6add1b01010, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=50.7 K 2023-04-05 09:55:57,808 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/60f15fbe71214421bb16044a09634adf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=48.3 K 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c460b88f2e35473dbc7ad6add1b01010, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 60f15fbe71214421bb16044a09634adf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting fe4413de39e74580930d17e4ccf80e3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680688555996 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 291aa7feb909460fa145c20c1084dd7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680688555996 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1ad490c302314e6ebe917ccce3c97962, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680688556626 2023-04-05 09:55:57,809 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 609537646fe84038bbbf6baf9b2b16f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680688556626 2023-04-05 09:55:57,810 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting de27c67557424755b8e20b040b4561f7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:55:57,810 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 57b2ef6c35c7427682944c06118223be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:55:57,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:57,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=53.67 KB heapSize=141.23 KB 2023-04-05 09:55:57,815 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:57,816 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#435 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:57,818 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#436 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:57,837 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/d41c95e50d9d49c983991fdc919b791f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d41c95e50d9d49c983991fdc919b791f 2023-04-05 09:55:57,840 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into d41c95e50d9d49c983991fdc919b791f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:57,840 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:57,840 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=12, startTime=1680688557807; duration=0sec 2023-04-05 09:55:57,841 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:55:57,841 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:55:57,841 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:55:57,842 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:55:57,842 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,842 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/7c5506cff0bc4ced9a24f859bae90eea, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=48.3 K 2023-04-05 09:55:57,843 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 7c5506cff0bc4ced9a24f859bae90eea, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680688555651 2023-04-05 09:55:57,844 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9a4d1d84560343b39d162d3a351ce055, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680688555996 2023-04-05 09:55:57,844 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting bf629865374e4d48a959d0113065a089, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680688556626 2023-04-05 09:55:57,844 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 412e11a94411443990427620b093c98e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:55:57,850 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:55:57,865 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/db9589f3f6ab4ca7ad29e88a74f6b6fb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db9589f3f6ab4ca7ad29e88a74f6b6fb 2023-04-05 09:55:57,870 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into db9589f3f6ab4ca7ad29e88a74f6b6fb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:57,870 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:57,870 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=12, startTime=1680688557807; duration=0sec 2023-04-05 09:55:57,871 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:57,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688617865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45182 deadline: 1680688617868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688557856, max timestamp=1680688557873Num uniques 859; 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688557862, max timestamp=1680688557873Num uniques 859; 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688557862, max timestamp=1680688557873Num uniques 859; 2023-04-05 09:55:57,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:55:57,900 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:57,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:57,903 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,905 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:57,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,907 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dfb3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:57,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:57,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:57,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:57,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:57,917 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:57,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:57,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:57,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:57,917 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:57,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,919 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:57,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,919 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:57,920 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:57,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:57,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:57,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:57,926 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,926 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:57,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:57,926 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:57,926 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:57,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:57,927 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:57,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688617937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688617943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688617975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688617975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,027 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,030 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,031 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,031 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,034 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae192dc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,037 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,040 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,042 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,042 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,042 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,043 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,044 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,045 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,047 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:58,047 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,047 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,047 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688618046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688618051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,147 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,151 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,152 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,152 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,153 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,153 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,154 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,154 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,155 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,155 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d5d0f0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,160 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,160 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,161 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,161 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,161 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,163 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,166 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,166 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,168 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,168 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,169 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,172 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:58,173 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,173 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688618183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688618183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/ffd44f8d2c7a4684bcdb7a968c35598d 2023-04-05 09:55:58,234 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e79faf9cf2124f8d9d859fa771c8acbf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e79faf9cf2124f8d9d859fa771c8acbf 2023-04-05 09:55:58,238 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into e79faf9cf2124f8d9d859fa771c8acbf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:55:58,238 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:58,238 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=12, startTime=1680688557807; duration=0sec 2023-04-05 09:55:58,238 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:55:58,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f14f89572b8e45ca818d2e66db309a20 2023-04-05 09:55:58,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688618253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688618255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,273 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,276 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,278 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,279 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,279 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,279 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,279 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,281 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@504bca83[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,284 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,284 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,284 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,288 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,290 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,290 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,292 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,294 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,298 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,299 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,299 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,299 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,299 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,399 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,403 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,409 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,410 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,410 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,411 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,411 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,412 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,412 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,412 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,413 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e11037e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,415 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,416 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,417 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,420 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,420 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,420 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,422 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,422 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,422 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,424 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,426 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:58,428 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,428 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688618487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688618488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,528 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,533 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,535 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,535 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,536 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,536 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,537 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,537 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,538 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,538 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,538 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,540 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c00512[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,545 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,548 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,548 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,550 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,551 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,552 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,554 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,554 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,555 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,555 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,555 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688618561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:58,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688618563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,655 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/30daf44fd6054c049988102ba8070594 2023-04-05 09:55:58,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,662 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,662 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,663 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/ffd44f8d2c7a4684bcdb7a968c35598d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d 2023-04-05 09:55:58,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:58,667 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:58,668 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d, entries=250, sequenceid=430, filesize=16.7 K 2023-04-05 09:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f14f89572b8e45ca818d2e66db309a20 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20 2023-04-05 09:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@349e10aa[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:58,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:58,671 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:58,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:58,673 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 09:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/30daf44fd6054c049988102ba8070594 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594 2023-04-05 09:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:58,676 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:58,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:58,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:58,676 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:58,676 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:58,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,685 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 09:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:58,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~161.11 KB/164976, currentSize=129.71 KB/132820 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 872ms, sequenceid=430, compaction requested=false 2023-04-05 09:55:58,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:58,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:58,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:58,692 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:58,692 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:58,691 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:58,691 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:58,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,692 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:58,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:58,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:58,792 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:58,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:58,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:58,795 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:58,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:58,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:58,795 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,797 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:58,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:58,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:58,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:58,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:58,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:58,799 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:55:58,817 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e9b2fe2a8b0f43fea5a933f875a3b91b 2023-04-05 09:55:59,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:55:59,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:55:59,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688619030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688619031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688619071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688619072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688619132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688619137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688619173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,182 DEBUG [Thread-2822] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:55:59,234 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/b01f3b3b3d7049c892f8d224bcd7f4ac 2023-04-05 09:55:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688619339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688619343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,642 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/488435f5956d487d83c20fab14c6079f 2023-04-05 09:55:59,646 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e9b2fe2a8b0f43fea5a933f875a3b91b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b 2023-04-05 09:55:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688619643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:55:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688619647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,650 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b, entries=150, sequenceid=469, filesize=11.9 K 2023-04-05 09:55:59,651 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/b01f3b3b3d7049c892f8d224bcd7f4ac as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac 2023-04-05 09:55:59,654 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac, entries=150, sequenceid=469, filesize=11.9 K 2023-04-05 09:55:59,654 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/488435f5956d487d83c20fab14c6079f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f 2023-04-05 09:55:59,657 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f, entries=150, sequenceid=469, filesize=11.9 K 2023-04-05 09:55:59,658 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 859ms, sequenceid=469, compaction requested=true 2023-04-05 09:55:59,658 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:59,658 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:59,663 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a0fcb68[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:59,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:59,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:59,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:59,667 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:59,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:59,670 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:59,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:59,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:59,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:59,670 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:59,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,672 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,673 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:59,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:59,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,678 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:59,678 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:59,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,778 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:59,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:59,781 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:59,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,783 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:59,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:59,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,783 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:59,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:59,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:59,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:59,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:59,785 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:59,785 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:55:59,786 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:55:59,794 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/421a6b15ba214e058dc3cc651aa811e1 2023-04-05 09:55:59,801 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f0f6ee0801d9445798c34e3389b25ecd 2023-04-05 09:55:59,812 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9b34faa1f4a54deea5f4ecfab90cf9ac 2023-04-05 09:55:59,815 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/421a6b15ba214e058dc3cc651aa811e1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1 2023-04-05 09:55:59,818 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 09:55:59,819 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f0f6ee0801d9445798c34e3389b25ecd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd 2023-04-05 09:55:59,821 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 09:55:59,822 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9b34faa1f4a54deea5f4ecfab90cf9ac as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac 2023-04-05 09:55:59,825 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 09:55:59,825 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 40ms, sequenceid=480, compaction requested=true 2023-04-05 09:55:59,825 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:59,825 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,827 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:59,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9335a4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:59,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:59,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:59,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,830 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,832 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:59,834 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:59,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:59,834 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:59,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,835 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,839 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:59,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:59,839 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,840 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,840 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:59,840 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,940 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:55:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:55:59,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:55:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:55:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:55:59,943 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:55:59,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:59,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:55:59,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:55:59,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,944 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,944 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:59,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:59,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:55:59,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:55:59,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:55:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:55:59,949 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:55:59,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c35db1[Count = 0] remaining members to acquire global barrier 2023-04-05 09:55:59,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:55:59,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,951 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:55:59,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:55:59,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:55:59,953 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:55:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:55:59,955 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:55:59,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:55:59,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:55:59,955 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:55:59,955 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:55:59,956 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:55:59,957 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:55:59,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:55:59,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:55:59,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:55:59,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:55:59,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:55:59,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:55:59,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:55:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:55:59,963 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:55:59,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,963 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:55:59,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,063 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,066 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,068 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,070 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,070 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,072 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,072 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,072 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,072 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,073 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509c04c3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,078 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,078 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,086 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,087 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,088 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:00,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:00,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,102 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,102 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,105 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:00,109 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,109 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,108 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e6a79074e792411abe1b5f558fa3fd56 2023-04-05 09:56:00,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/ed549f528ca444d0be25f3213c0ba88d 2023-04-05 09:56:00,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/492977eddbd64e29b702abbca682fab1 2023-04-05 09:56:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560149, max timestamp=1680688560166Num uniques 859; 2023-04-05 09:56:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560149, max timestamp=1680688560166Num uniques 859; 2023-04-05 09:56:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e6a79074e792411abe1b5f558fa3fd56 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56 2023-04-05 09:56:00,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560149, max timestamp=1680688560166Num uniques 859; 2023-04-05 09:56:00,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56, entries=200, sequenceid=492, filesize=14.3 K 2023-04-05 09:56:00,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/ed549f528ca444d0be25f3213c0ba88d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d 2023-04-05 09:56:00,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d, entries=100, sequenceid=492, filesize=9.5 K 2023-04-05 09:56:00,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/492977eddbd64e29b702abbca682fab1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1 2023-04-05 09:56:00,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1, entries=100, sequenceid=492, filesize=9.5 K 2023-04-05 09:56:00,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=53.67 KB/54960 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 84ms, sequenceid=492, compaction requested=true 2023-04-05 09:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,182 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:56:00,184 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:00,184 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 69098 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:00,184 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:56:00,184 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,184 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e79faf9cf2124f8d9d859fa771c8acbf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=67.5 K 2023-04-05 09:56:00,185 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e79faf9cf2124f8d9d859fa771c8acbf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:56:00,185 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ffd44f8d2c7a4684bcdb7a968c35598d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680688557795 2023-04-05 09:56:00,186 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:00,186 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:56:00,186 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,186 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d41c95e50d9d49c983991fdc919b791f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=57.9 K 2023-04-05 09:56:00,186 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e9b2fe2a8b0f43fea5a933f875a3b91b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680688557935 2023-04-05 09:56:00,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=60.38 KB heapSize=161.16 KB 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,187 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d41c95e50d9d49c983991fdc919b791f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,187 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 421a6b15ba214e058dc3cc651aa811e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680688559008 2023-04-05 09:56:00,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:00,187 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f14f89572b8e45ca818d2e66db309a20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680688557802 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,189 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b01f3b3b3d7049c892f8d224bcd7f4ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680688557935 2023-04-05 09:56:00,189 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e6a79074e792411abe1b5f558fa3fd56, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688560090 2023-04-05 09:56:00,190 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f0f6ee0801d9445798c34e3389b25ecd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680688559008 2023-04-05 09:56:00,191 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ed549f528ca444d0be25f3213c0ba88d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688560097 2023-04-05 09:56:00,209 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,214 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,220 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,220 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,220 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,220 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,220 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,220 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,227 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,227 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,227 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,227 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,227 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/16b63ef730a7464694cc618f4758acc8 2023-04-05 09:56:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,245 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,245 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,245 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,245 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,247 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560209, max timestamp=1680688560247Num uniques 859; 2023-04-05 09:56:00,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560209, max timestamp=1680688560247Num uniques 859; 2023-04-05 09:56:00,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680688560209, max timestamp=1680688560247Num uniques 859; 2023-04-05 09:56:00,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 09:56:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/11fd99aeb4c045f5ae25b8e1cd2779f4 2023-04-05 09:56:00,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@148cd133[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,253 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,253 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,256 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/1555a508130e4b34a69efcda71ac7301 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1555a508130e4b34a69efcda71ac7301 2023-04-05 09:56:00,256 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,261 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 1555a508130e4b34a69efcda71ac7301(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:00,261 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,261 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,261 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,261 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,262 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=11, startTime=1680688560182; duration=0sec 2023-04-05 09:56:00,262 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,262 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:00,262 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:00,263 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:00,263 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:56:00,263 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,263 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db9589f3f6ab4ca7ad29e88a74f6b6fb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=57.9 K 2023-04-05 09:56:00,264 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting db9589f3f6ab4ca7ad29e88a74f6b6fb, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680688557754 2023-04-05 09:56:00,264 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 30daf44fd6054c049988102ba8070594, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680688557802 2023-04-05 09:56:00,265 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 488435f5956d487d83c20fab14c6079f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680688557935 2023-04-05 09:56:00,265 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9b34faa1f4a54deea5f4ecfab90cf9ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680688559008 2023-04-05 09:56:00,265 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 492977eddbd64e29b702abbca682fab1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688560097 2023-04-05 09:56:00,266 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,266 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,266 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,268 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,271 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,271 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:00,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/36ea3bdef1bc4f179b6561e01256e4e0 2023-04-05 09:56:00,278 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/16b63ef730a7464694cc618f4758acc8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8 2023-04-05 09:56:00,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8, entries=250, sequenceid=517, filesize=16.7 K 2023-04-05 09:56:00,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/11fd99aeb4c045f5ae25b8e1cd2779f4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4 2023-04-05 09:56:00,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4, entries=150, sequenceid=517, filesize=11.9 K 2023-04-05 09:56:00,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/36ea3bdef1bc4f179b6561e01256e4e0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0 2023-04-05 09:56:00,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0, entries=150, sequenceid=517, filesize=11.9 K 2023-04-05 09:56:00,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~161.11 KB/164976, currentSize=100.63 KB/103050 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 108ms, sequenceid=517, compaction requested=false 2023-04-05 09:56:00,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,297 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/f7ca7b967c2b4f848aa05e4204ca2ae0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f7ca7b967c2b4f848aa05e4204ca2ae0 2023-04-05 09:56:00,300 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into f7ca7b967c2b4f848aa05e4204ca2ae0(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:00,300 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,300 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=11, startTime=1680688560182; duration=0sec 2023-04-05 09:56:00,301 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:00,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=107.34 KB heapSize=300.61 KB 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:00,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:00,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/829ddbb6603b4e5ba7a59b62bc20822a 2023-04-05 09:56:00,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688620317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688620317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/e1739b64db19414bb5fd52fa74a0a383 2023-04-05 09:56:00,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688620322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688620327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,371 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,374 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,374 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,375 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,375 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,376 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,376 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,379 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,379 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,379 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,379 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,381 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2187f813[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,383 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,385 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,385 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,385 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,385 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,386 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,386 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,388 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,389 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,389 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,389 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,389 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,390 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,390 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,391 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,393 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,394 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,394 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,394 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688620428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688620428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688620436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688620442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,494 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,497 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,499 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,499 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,500 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa70747[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,504 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,504 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,505 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,505 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,506 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,510 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,512 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,512 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,512 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,514 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,515 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,516 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,517 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,517 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,517 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,517 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:00,617 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,621 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,623 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,623 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,623 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,623 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,625 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,626 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,626 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,626 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,627 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fada46c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,630 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,632 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,635 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,635 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,638 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,640 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,641 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,641 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688620637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688620637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688620642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,652 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/89f491abc1694f359f7e1752fd87de71 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/89f491abc1694f359f7e1752fd87de71 2023-04-05 09:56:00,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688620649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,656 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 89f491abc1694f359f7e1752fd87de71(size=12.9 K), total size for store is 29.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:00,656 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,656 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=11, startTime=1680688560182; duration=0sec 2023-04-05 09:56:00,656 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:00,741 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/8208b9ef1bd24109a2ed82de682b5ffd 2023-04-05 09:56:00,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,744 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,745 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,746 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,746 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/829ddbb6603b4e5ba7a59b62bc20822a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a 2023-04-05 09:56:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,748 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,749 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:00,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a, entries=250, sequenceid=551, filesize=16.7 K 2023-04-05 09:56:00,751 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/e1739b64db19414bb5fd52fa74a0a383 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383 2023-04-05 09:56:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:00,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddafbe3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:00,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:00,753 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:00,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:00,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383, entries=150, sequenceid=551, filesize=11.9 K 2023-04-05 09:56:00,756 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/8208b9ef1bd24109a2ed82de682b5ffd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd 2023-04-05 09:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:00,758 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:00,758 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:00,758 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:00,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:00,759 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:00,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd, entries=150, sequenceid=551, filesize=11.9 K 2023-04-05 09:56:00,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~300.56 KB/307776, currentSize=73.80 KB/75570 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 457ms, sequenceid=551, compaction requested=true 2023-04-05 09:56:00,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,760 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:00,760 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:00,760 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:00,760 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:00,760 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:00,760 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,760 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:00,760 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:00,761 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,761 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:56:00,761 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,761 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/89f491abc1694f359f7e1752fd87de71, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=46.2 K 2023-04-05 09:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:00,761 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 89f491abc1694f359f7e1752fd87de71, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688559008 2023-04-05 09:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:00,761 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:56:00,762 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,762 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1555a508130e4b34a69efcda71ac7301, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=36.7 K 2023-04-05 09:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:00,763 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 1555a508130e4b34a69efcda71ac7301, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688559008 2023-04-05 09:56:00,763 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,763 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 16b63ef730a7464694cc618f4758acc8, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680688560163 2023-04-05 09:56:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:00,763 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 11fd99aeb4c045f5ae25b8e1cd2779f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680688560174 2023-04-05 09:56:00,763 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 829ddbb6603b4e5ba7a59b62bc20822a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560287 2023-04-05 09:56:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,763 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e1739b64db19414bb5fd52fa74a0a383, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560288 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:00,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:00,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:00,767 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:00,767 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,767 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:00,769 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,770 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#460 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,785 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/4ab016c6d39f4becbf925cb79a469ab4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4ab016c6d39f4becbf925cb79a469ab4 2023-04-05 09:56:00,785 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/e4c0b44f84cb43f180aa23c3ba89f54b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e4c0b44f84cb43f180aa23c3ba89f54b 2023-04-05 09:56:00,789 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 4ab016c6d39f4becbf925cb79a469ab4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:00,789 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into e4c0b44f84cb43f180aa23c3ba89f54b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:00,789 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,789 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:00,789 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=13, startTime=1680688560760; duration=0sec 2023-04-05 09:56:00,789 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=13, startTime=1680688560760; duration=0sec 2023-04-05 09:56:00,790 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:00,790 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:00,790 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:00,790 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:00,790 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:56:00,790 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,791 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f7ca7b967c2b4f848aa05e4204ca2ae0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=36.7 K 2023-04-05 09:56:00,791 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting f7ca7b967c2b4f848aa05e4204ca2ae0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680688559008 2023-04-05 09:56:00,791 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 36ea3bdef1bc4f179b6561e01256e4e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680688560174 2023-04-05 09:56:00,792 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 8208b9ef1bd24109a2ed82de682b5ffd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560288 2023-04-05 09:56:00,796 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#461 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:00,867 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:00,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:00,870 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:00,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:00,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:00,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,871 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,872 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:00,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:00,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,872 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:00,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:00,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:00,874 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:00,874 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:00,874 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:00,881 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/4f4fbf64963f46c1b1f380423e0b5d03 2023-04-05 09:56:00,889 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/d47dd2a4d3f749fb8ae9253ae3ab470a 2023-04-05 09:56:00,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:00,946 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:00,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688620963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688620965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688620967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:00,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:00,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688620968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688621069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688621070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688621071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688621075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,205 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/de5e72000f28461eb19ab356780657da as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/de5e72000f28461eb19ab356780657da 2023-04-05 09:56:01,209 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into de5e72000f28461eb19ab356780657da(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:01,209 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:01,209 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=13, startTime=1680688560760; duration=0sec 2023-04-05 09:56:01,209 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:01,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688621278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688621279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688621280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688621280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,299 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9bc30acab0d84b51a627e84c22dfb5f4 2023-04-05 09:56:01,303 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/4f4fbf64963f46c1b1f380423e0b5d03 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03 2023-04-05 09:56:01,306 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03, entries=150, sequenceid=568, filesize=11.9 K 2023-04-05 09:56:01,306 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/d47dd2a4d3f749fb8ae9253ae3ab470a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a 2023-04-05 09:56:01,310 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a, entries=150, sequenceid=568, filesize=11.9 K 2023-04-05 09:56:01,310 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9bc30acab0d84b51a627e84c22dfb5f4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4 2023-04-05 09:56:01,313 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4, entries=150, sequenceid=568, filesize=11.9 K 2023-04-05 09:56:01,314 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 440ms, sequenceid=568, compaction requested=false 2023-04-05 09:56:01,314 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:01,314 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:01,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:01,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:01,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7383c12b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:01,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:01,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:01,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:01,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:01,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:01,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,323 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:01,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:01,325 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:01,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:01,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:01,326 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:01,326 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:01,327 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,327 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:01,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,327 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:01,328 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,329 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:01,334 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:01,334 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:01,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:01,335 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:01,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:01,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:01,335 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:01,335 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:01,335 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:01,435 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:01,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:01,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:01,438 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:01,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:01,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:01,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:01,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,440 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:01,442 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:01,442 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:01,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:01,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:01,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:01,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,444 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:01,444 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:01,444 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:01,444 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:01,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688621600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688621604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688621605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688621605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688621706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688621706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688621710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688621710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,854 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/6622c21cbe0946aab8581ed07a8e3134 2023-04-05 09:56:01,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688621910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688621910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688621914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688621914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688622216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688622217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688622218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688622220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,263 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f936bf44de734ffd86be485129d7d836 2023-04-05 09:56:02,271 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4ef6764ed1d449cebacba822bb95e30b 2023-04-05 09:56:02,274 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/6622c21cbe0946aab8581ed07a8e3134 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134 2023-04-05 09:56:02,277 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134, entries=150, sequenceid=587, filesize=11.9 K 2023-04-05 09:56:02,278 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/f936bf44de734ffd86be485129d7d836 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836 2023-04-05 09:56:02,281 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836, entries=150, sequenceid=587, filesize=11.9 K 2023-04-05 09:56:02,281 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4ef6764ed1d449cebacba822bb95e30b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b 2023-04-05 09:56:02,284 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b, entries=150, sequenceid=587, filesize=11.9 K 2023-04-05 09:56:02,285 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 841ms, sequenceid=587, compaction requested=true 2023-04-05 09:56:02,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:02,285 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:02,286 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,286 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,287 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64097d23[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:02,288 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:02,288 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:02,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,290 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:02,291 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:02,294 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:02,294 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:02,294 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:02,294 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:02,294 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:02,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,296 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:02,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,298 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:02,300 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:02,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:02,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:02,301 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,301 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:02,301 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,401 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:02,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:02,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:02,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:02,404 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:02,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:02,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:02,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:02,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:02,406 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:02,406 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,406 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:02,407 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:02,408 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:02,408 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:02,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:02,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688622748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688622749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688622750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688622751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,818 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/b26b222c2f1b4305beaab3d5f1380e2e 2023-04-05 09:56:02,826 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/340dcb263f554b978f4d9a5a104c29c5 2023-04-05 09:56:02,835 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/e850b4a270c7415eab7e0e73f4f73b83 2023-04-05 09:56:02,840 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/b26b222c2f1b4305beaab3d5f1380e2e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e 2023-04-05 09:56:02,843 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e, entries=150, sequenceid=601, filesize=11.9 K 2023-04-05 09:56:02,844 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/340dcb263f554b978f4d9a5a104c29c5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5 2023-04-05 09:56:02,847 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5, entries=150, sequenceid=601, filesize=11.9 K 2023-04-05 09:56:02,848 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/e850b4a270c7415eab7e0e73f4f73b83 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83 2023-04-05 09:56:02,850 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83, entries=150, sequenceid=601, filesize=11.9 K 2023-04-05 09:56:02,851 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 443ms, sequenceid=601, compaction requested=true 2023-04-05 09:56:02,851 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:02,851 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:02,854 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,854 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:02,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:02,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16752b0a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:02,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:02,856 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,858 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,858 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:02,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:02,859 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:02,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:02,860 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:02,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:02,863 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:02,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:02,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:02,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:02,863 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:02,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/36c5f44d905a4e0b9d0158b7148bad6b 2023-04-05 09:56:02,864 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:02,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:02,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:02,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:02,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:02,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:02,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:02,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:02,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:02,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:02,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:02,870 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:02,870 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:02,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/be27949f36934995881e93247b9ccce0 2023-04-05 09:56:02,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688622874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688622874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4333a47e3efa4e208ef0b4530114de73 2023-04-05 09:56:02,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688622882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:02,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688622883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/36c5f44d905a4e0b9d0158b7148bad6b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b 2023-04-05 09:56:02,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b, entries=200, sequenceid=620, filesize=14.3 K 2023-04-05 09:56:02,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/be27949f36934995881e93247b9ccce0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0 2023-04-05 09:56:02,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0, entries=150, sequenceid=620, filesize=11.9 K 2023-04-05 09:56:02,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/4333a47e3efa4e208ef0b4530114de73 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73 2023-04-05 09:56:02,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73, entries=150, sequenceid=620, filesize=11.9 K 2023-04-05 09:56:02,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 72ms, sequenceid=620, compaction requested=true 2023-04-05 09:56:02,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:02,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:02,927 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:02,928 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:02,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:02,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:02,929 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64494 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:02,929 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:02,929 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/A is initiating minor compaction (all files) 2023-04-05 09:56:02,929 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/B is initiating minor compaction (all files) 2023-04-05 09:56:02,929 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/A in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,929 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/B in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,929 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4ab016c6d39f4becbf925cb79a469ab4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=63.0 K 2023-04-05 09:56:02,930 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e4c0b44f84cb43f180aa23c3ba89f54b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=60.6 K 2023-04-05 09:56:02,930 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4ab016c6d39f4becbf925cb79a469ab4, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560288 2023-04-05 09:56:02,930 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting e4c0b44f84cb43f180aa23c3ba89f54b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560288 2023-04-05 09:56:02,930 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4f4fbf64963f46c1b1f380423e0b5d03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680688560314 2023-04-05 09:56:02,930 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d47dd2a4d3f749fb8ae9253ae3ab470a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680688560314 2023-04-05 09:56:02,931 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6622c21cbe0946aab8581ed07a8e3134, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680688560962 2023-04-05 09:56:02,931 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f936bf44de734ffd86be485129d7d836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680688560962 2023-04-05 09:56:02,931 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b26b222c2f1b4305beaab3d5f1380e2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680688561599 2023-04-05 09:56:02,932 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 340dcb263f554b978f4d9a5a104c29c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680688561599 2023-04-05 09:56:02,932 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 36c5f44d905a4e0b9d0158b7148bad6b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680688562747 2023-04-05 09:56:02,933 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting be27949f36934995881e93247b9ccce0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680688562747 2023-04-05 09:56:02,942 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#A#compaction#474 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:02,950 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#B#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:02,960 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/82dbef88ae7e482d9a15c77ef01eb030 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/82dbef88ae7e482d9a15c77ef01eb030 2023-04-05 09:56:02,970 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:02,970 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/A of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 82dbef88ae7e482d9a15c77ef01eb030(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:02,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:02,971 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:02,971 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/A, priority=11, startTime=1680688562927; duration=0sec 2023-04-05 09:56:02,971 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:02,971 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:02,974 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:02,974 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): 62afd6a1e0fb2ca72b38ec7d7142b5cc/C is initiating minor compaction (all files) 2023-04-05 09:56:02,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:02,974 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62afd6a1e0fb2ca72b38ec7d7142b5cc/C in TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,975 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/de5e72000f28461eb19ab356780657da, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp, totalSize=60.6 K 2023-04-05 09:56:02,975 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting de5e72000f28461eb19ab356780657da, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680688560288 2023-04-05 09:56:02,975 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9bc30acab0d84b51a627e84c22dfb5f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680688560314 2023-04-05 09:56:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:02,976 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:02,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:02,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:02,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:02,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,979 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:02,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:02,979 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:02,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:02,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:02,984 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4ef6764ed1d449cebacba822bb95e30b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680688560962 2023-04-05 09:56:02,984 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e850b4a270c7415eab7e0e73f4f73b83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680688561599 2023-04-05 09:56:02,985 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4333a47e3efa4e208ef0b4530114de73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680688562747 2023-04-05 09:56:02,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:02,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:02,985 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:02,985 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:02,986 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:02,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:02,998 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62afd6a1e0fb2ca72b38ec7d7142b5cc#C#compaction#477 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:03,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680688623205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,214 DEBUG [Thread-2822] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., hostname=jenkins-hbase4.apache.org,46827,1680688416991, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 09:56:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,369 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/fa244bb65d8043a68e415c8f0bfbb858 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fa244bb65d8043a68e415c8f0bfbb858 2023-04-05 09:56:03,373 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/B of 62afd6a1e0fb2ca72b38ec7d7142b5cc into fa244bb65d8043a68e415c8f0bfbb858(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:03,373 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:03,373 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/B, priority=11, startTime=1680688562927; duration=0sec 2023-04-05 09:56:03,373 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:03,422 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e0b474a7c8f64f279054c53eb743d8c1 2023-04-05 09:56:03,426 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/332eb39de8c349f3911735a9f6c8f66b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/332eb39de8c349f3911735a9f6c8f66b 2023-04-05 09:56:03,431 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 62afd6a1e0fb2ca72b38ec7d7142b5cc/C of 62afd6a1e0fb2ca72b38ec7d7142b5cc into 332eb39de8c349f3911735a9f6c8f66b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:03,431 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:03,431 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc., storeName=62afd6a1e0fb2ca72b38ec7d7142b5cc/C, priority=11, startTime=1680688562928; duration=0sec 2023-04-05 09:56:03,431 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:03,432 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/4a0ab06331324b09afa216a25c57c740 2023-04-05 09:56:03,442 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/0756b3b3200946bb9aee11643affe331 2023-04-05 09:56:03,451 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/e0b474a7c8f64f279054c53eb743d8c1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e0b474a7c8f64f279054c53eb743d8c1 2023-04-05 09:56:03,454 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e0b474a7c8f64f279054c53eb743d8c1, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:56:03,455 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/4a0ab06331324b09afa216a25c57c740 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4a0ab06331324b09afa216a25c57c740 2023-04-05 09:56:03,457 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4a0ab06331324b09afa216a25c57c740, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:56:03,458 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/0756b3b3200946bb9aee11643affe331 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0756b3b3200946bb9aee11643affe331 2023-04-05 09:56:03,460 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0756b3b3200946bb9aee11643affe331, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 09:56:03,461 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 475ms, sequenceid=634, compaction requested=false 2023-04-05 09:56:03,461 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:03,461 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:03,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:03,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:03,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:03,461 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:03,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:03,463 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,464 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c4acfc4[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:03,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:03,466 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,466 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:03,467 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,467 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:03,468 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:03,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:03,468 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:03,470 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:03,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:03,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:03,470 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:03,470 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:03,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:03,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,473 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:03,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:03,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:03,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:03,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:03,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:03,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:03,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:03,479 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:03,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:03,479 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:03,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:03,579 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:03,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:03,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:03,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:03,595 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:03,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:03,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:03,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:03,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,595 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:03,596 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:03,596 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:03,597 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,597 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:03,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:03,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:03,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:03,598 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:03,598 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:03,598 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:03,598 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:03,599 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:03,599 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:03,599 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:03,599 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:03,599 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:03,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:03,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688623977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688623977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688623978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:03,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688623983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,006 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1911cbaa6eaf46e4835844eb5686d421 2023-04-05 09:56:04,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:04,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45140 deadline: 1680688624282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:04,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45216 deadline: 1680688624283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:04,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45182 deadline: 1680688624284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:04,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45180 deadline: 1680688624288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,415 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/73edc304f956409eae03259bef8e13f3 2023-04-05 09:56:04,423 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9dda6c10005948d8941a3bd9bea7503f 2023-04-05 09:56:04,427 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/1911cbaa6eaf46e4835844eb5686d421 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1911cbaa6eaf46e4835844eb5686d421 2023-04-05 09:56:04,430 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1911cbaa6eaf46e4835844eb5686d421, entries=150, sequenceid=656, filesize=11.9 K 2023-04-05 09:56:04,430 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/73edc304f956409eae03259bef8e13f3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73edc304f956409eae03259bef8e13f3 2023-04-05 09:56:04,438 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73edc304f956409eae03259bef8e13f3, entries=150, sequenceid=656, filesize=11.9 K 2023-04-05 09:56:04,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/9dda6c10005948d8941a3bd9bea7503f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9dda6c10005948d8941a3bd9bea7503f 2023-04-05 09:56:04,442 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9dda6c10005948d8941a3bd9bea7503f, entries=150, sequenceid=656, filesize=11.9 K 2023-04-05 09:56:04,443 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 845ms, sequenceid=656, compaction requested=true 2023-04-05 09:56:04,443 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:04,443 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:04,445 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412dd9ff[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:04,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:04,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,448 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:04,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:04,449 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:04,449 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,449 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:04,450 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:04,452 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:04,452 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:04,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:04,454 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:04,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:04,455 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:04,459 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,459 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:04,460 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:04,460 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:04,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:04,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:04,460 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:04,460 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:04,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:04,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:04,560 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:04,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:04,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:04,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:04,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:04,563 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:04,563 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:04,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:04,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:04,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,564 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:04,565 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:04,565 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:04,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:04,566 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:04,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. started... 2023-04-05 09:56:04,567 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:04,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:04,568 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:04,568 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:04,581 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/96814baef27e487fb3cb7be0168e99d8 2023-04-05 09:56:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:04,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. as already flushing 2023-04-05 09:56:04,857 DEBUG [Thread-2829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:62753 2023-04-05 09:56:04,857 DEBUG [Thread-2829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,859 DEBUG [Thread-2833] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:62753 2023-04-05 09:56:04,859 DEBUG [Thread-2833] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,874 DEBUG [Thread-2814] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:62753 2023-04-05 09:56:04,874 DEBUG [Thread-2818] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:62753 2023-04-05 09:56:04,874 DEBUG [Thread-2816] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:62753 2023-04-05 09:56:04,874 DEBUG [Thread-2818] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,874 DEBUG [Thread-2814] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,874 DEBUG [Thread-2816] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,875 DEBUG [Thread-2820] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:62753 2023-04-05 09:56:04,875 DEBUG [Thread-2820] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,882 DEBUG [Thread-2825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:62753 2023-04-05 09:56:04,882 DEBUG [Thread-2825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,883 DEBUG [Thread-2831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:62753 2023-04-05 09:56:04,883 DEBUG [Thread-2827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:62753 2023-04-05 09:56:04,883 DEBUG [Thread-2831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,883 DEBUG [Thread-2827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:04,993 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/a18a3975af7648509d04e4130a6f406a 2023-04-05 09:56:05,401 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/2bb611c4fdcb4d8291051e97641f0503 2023-04-05 09:56:05,405 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/96814baef27e487fb3cb7be0168e99d8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/96814baef27e487fb3cb7be0168e99d8 2023-04-05 09:56:05,408 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/96814baef27e487fb3cb7be0168e99d8, entries=150, sequenceid=668, filesize=11.9 K 2023-04-05 09:56:05,409 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/a18a3975af7648509d04e4130a6f406a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a18a3975af7648509d04e4130a6f406a 2023-04-05 09:56:05,412 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a18a3975af7648509d04e4130a6f406a, entries=150, sequenceid=668, filesize=11.9 K 2023-04-05 09:56:05,413 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/2bb611c4fdcb4d8291051e97641f0503 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/2bb611c4fdcb4d8291051e97641f0503 2023-04-05 09:56:05,416 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/2bb611c4fdcb4d8291051e97641f0503, entries=150, sequenceid=668, filesize=11.9 K 2023-04-05 09:56:05,417 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 850ms, sequenceid=668, compaction requested=true 2023-04-05 09:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:05,420 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e62449c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:05,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:05,422 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,423 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:05,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,423 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:05,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:05,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:05,425 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:05,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:05,428 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:05,428 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:05,428 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:05,428 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:05,428 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:05,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:05,430 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,430 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:05,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,433 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:05,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:05,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:05,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:05,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:05,437 DEBUG [Thread-2824] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,437 DEBUG [Thread-2824] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:05,437 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:05,537 DEBUG [Thread-2824] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:05,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:13,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:13,252 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62afd6a1e0fb2ca72b38ec7d7142b5cc 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=A 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=B 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62afd6a1e0fb2ca72b38ec7d7142b5cc, store=C 2023-04-05 09:56:13,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:13,253 DEBUG [Thread-2822] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:62753 2023-04-05 09:56:13,253 DEBUG [Thread-2822] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:13,254 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 105 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 197 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 95 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2120 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6355 rows 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2112 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6336 rows 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2121 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6359 rows 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2122 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6366 rows 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(402): scanned 2109 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(403): verified 6326 rows 2023-04-05 09:56:13,256 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:56:13,257 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df63a4 to 127.0.0.1:62753 2023-04-05 09:56:13,257 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:13,260 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:56:13,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:56:13,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:13,267 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688573266"}]},"ts":"1680688573266"} 2023-04-05 09:56:13,267 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:56:13,269 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:56:13,270 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, UNASSIGN}] 2023-04-05 09:56:13,273 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, UNASSIGN 2023-04-05 09:56:13,273 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=62afd6a1e0fb2ca72b38ec7d7142b5cc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:13,274 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688573273"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688573273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688573273"}]},"ts":"1680688573273"} 2023-04-05 09:56:13,275 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:56:13,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:13,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:13,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 62afd6a1e0fb2ca72b38ec7d7142b5cc, disabling compactions & flushes 2023-04-05 09:56:13,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:13,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/28d4e9a7734b40009e245ee8600429b3 2023-04-05 09:56:13,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:14,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/ace76080bbb747a2b88ecc4289cd1eb9 2023-04-05 09:56:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:14,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/db8ab576e3c04c23b80c954819ddafe8 2023-04-05 09:56:14,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/A/28d4e9a7734b40009e245ee8600429b3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/28d4e9a7734b40009e245ee8600429b3 2023-04-05 09:56:14,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/28d4e9a7734b40009e245ee8600429b3, entries=150, sequenceid=689, filesize=11.9 K 2023-04-05 09:56:14,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/B/ace76080bbb747a2b88ecc4289cd1eb9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ace76080bbb747a2b88ecc4289cd1eb9 2023-04-05 09:56:14,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ace76080bbb747a2b88ecc4289cd1eb9, entries=150, sequenceid=689, filesize=11.9 K 2023-04-05 09:56:14,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/.tmp/C/db8ab576e3c04c23b80c954819ddafe8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db8ab576e3c04c23b80c954819ddafe8 2023-04-05 09:56:14,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db8ab576e3c04c23b80c954819ddafe8, entries=150, sequenceid=689, filesize=11.9 K 2023-04-05 09:56:14,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=0 B/0 for 62afd6a1e0fb2ca72b38ec7d7142b5cc in 1267ms, sequenceid=689, compaction requested=true 2023-04-05 09:56:14,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:14,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:14,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:14,519 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. because compaction request was cancelled 2023-04-05 09:56:14,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:14,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:14,519 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. because compaction request was cancelled 2023-04-05 09:56:14,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. after waiting 0 ms 2023-04-05 09:56:14,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:14,519 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. because compaction request was cancelled 2023-04-05 09:56:14,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:14,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/eb2d5bebd87b49549ec0a1f70cf1005c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/f56a1019e05e47508b28c99307caf0bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a986652e78fe41b5a63d2a4c1a581cd8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/9ab771643cee4f25b79a85b13ccf8472, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/20b697b9d5564cacb7ed5331a8415526, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c460b88f2e35473dbc7ad6add1b01010, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e79faf9cf2124f8d9d859fa771c8acbf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/89f491abc1694f359f7e1752fd87de71, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4ab016c6d39f4becbf925cb79a469ab4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b] to archive 2023-04-05 09:56:14,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:14,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1bea23dc71184e339f4bf1308b6d354f 2023-04-05 09:56:14,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a95a4b52edc44ef3a50916f4c936b3a4 2023-04-05 09:56:14,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c2a0ecddca5a4a65b4d51cfebb7e7cdd 2023-04-05 09:56:14,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/eb2d5bebd87b49549ec0a1f70cf1005c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/eb2d5bebd87b49549ec0a1f70cf1005c 2023-04-05 09:56:14,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8f6464d4f2bd45c89cfe5d3008641f44 2023-04-05 09:56:14,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/f56a1019e05e47508b28c99307caf0bc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/f56a1019e05e47508b28c99307caf0bc 2023-04-05 09:56:14,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/3153aa8210f44681af2191d22c45b483 2023-04-05 09:56:14,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/88ac8e3ec883435197f6fcdd4be07cfc 2023-04-05 09:56:14,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/8c0000414a5b457b82cec33cf9e53267 2023-04-05 09:56:14,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a986652e78fe41b5a63d2a4c1a581cd8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/a986652e78fe41b5a63d2a4c1a581cd8 2023-04-05 09:56:14,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/01f1d58d65394cbd86c5323b43b67029 2023-04-05 09:56:14,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6d4e719c8e8549cbab46c9b9934e308f 2023-04-05 09:56:14,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c8a1733e7d9a4f4f9144323ee98bc8d8 2023-04-05 09:56:14,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/9ab771643cee4f25b79a85b13ccf8472 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/9ab771643cee4f25b79a85b13ccf8472 2023-04-05 09:56:14,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/2a0e70665b2b48129652a69d9893e8fa 2023-04-05 09:56:14,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cbd1232496844538854da037b2639aaf 2023-04-05 09:56:14,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ceab9ee3d636485d9196aed228da8857 2023-04-05 09:56:14,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/20b697b9d5564cacb7ed5331a8415526 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/20b697b9d5564cacb7ed5331a8415526 2023-04-05 09:56:14,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/610bdef38aff4da9a911fdfae3e6c427 2023-04-05 09:56:14,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/d15b5d7adfc54619b279a73196d90e4a 2023-04-05 09:56:14,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c460b88f2e35473dbc7ad6add1b01010 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/c460b88f2e35473dbc7ad6add1b01010 2023-04-05 09:56:14,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/cb756e2b42d24549a75106b5e7035c57 2023-04-05 09:56:14,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/fe4413de39e74580930d17e4ccf80e3b 2023-04-05 09:56:14,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1ad490c302314e6ebe917ccce3c97962 2023-04-05 09:56:14,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/de27c67557424755b8e20b040b4561f7 2023-04-05 09:56:14,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e79faf9cf2124f8d9d859fa771c8acbf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e79faf9cf2124f8d9d859fa771c8acbf 2023-04-05 09:56:14,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/ffd44f8d2c7a4684bcdb7a968c35598d 2023-04-05 09:56:14,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e9b2fe2a8b0f43fea5a933f875a3b91b 2023-04-05 09:56:14,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/421a6b15ba214e058dc3cc651aa811e1 2023-04-05 09:56:14,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e6a79074e792411abe1b5f558fa3fd56 2023-04-05 09:56:14,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/89f491abc1694f359f7e1752fd87de71 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/89f491abc1694f359f7e1752fd87de71 2023-04-05 09:56:14,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/16b63ef730a7464694cc618f4758acc8 2023-04-05 09:56:14,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/829ddbb6603b4e5ba7a59b62bc20822a 2023-04-05 09:56:14,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4ab016c6d39f4becbf925cb79a469ab4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4ab016c6d39f4becbf925cb79a469ab4 2023-04-05 09:56:14,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/4f4fbf64963f46c1b1f380423e0b5d03 2023-04-05 09:56:14,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/6622c21cbe0946aab8581ed07a8e3134 2023-04-05 09:56:14,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/b26b222c2f1b4305beaab3d5f1380e2e 2023-04-05 09:56:14,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/36c5f44d905a4e0b9d0158b7148bad6b 2023-04-05 09:56:14,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/516d62310f3942c4bd36adb81895723a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/107e96cbd7d641819c246e9752cf6408, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/39622020f3534a73a0dda8fe9635e88a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73803ee68807437397a34800b280a29f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a2777c5401a6426c9bfc981f217417d0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/60f15fbe71214421bb16044a09634adf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d41c95e50d9d49c983991fdc919b791f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1555a508130e4b34a69efcda71ac7301, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e4c0b44f84cb43f180aa23c3ba89f54b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0] to archive 2023-04-05 09:56:14,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:14,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d0abf23e016d482f99f761d2403a541e 2023-04-05 09:56:14,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/6e0f9da122c0444ba57c8173841232a7 2023-04-05 09:56:14,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/516d62310f3942c4bd36adb81895723a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/516d62310f3942c4bd36adb81895723a 2023-04-05 09:56:14,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/35043553249f40919142fe900ac50c6c 2023-04-05 09:56:14,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/997040c7c28e4b59aacff90cb45db939 2023-04-05 09:56:14,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/107e96cbd7d641819c246e9752cf6408 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/107e96cbd7d641819c246e9752cf6408 2023-04-05 09:56:14,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e7e2f61335a84b55b44f8a06dc57935a 2023-04-05 09:56:14,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/24c393b4e0774a4599f5212be3ae85c5 2023-04-05 09:56:14,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/0051f91164d148bfacb948ee0025983c 2023-04-05 09:56:14,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/39622020f3534a73a0dda8fe9635e88a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/39622020f3534a73a0dda8fe9635e88a 2023-04-05 09:56:14,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1aa4806a3bd14e10a506f64038a91a8b 2023-04-05 09:56:14,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/600c6102571247c2aa046d2fbcdb7446 2023-04-05 09:56:14,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73803ee68807437397a34800b280a29f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73803ee68807437397a34800b280a29f 2023-04-05 09:56:14,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4c4a5685e113442b8209258dc43174b3 2023-04-05 09:56:14,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b148f700255847b0b9312bedf1f36a80 2023-04-05 09:56:14,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/847b444bd3bf42d4b229d5f867ef6f11 2023-04-05 09:56:14,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/26f8240f4d734a03bf484531ff0106c8 2023-04-05 09:56:14,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a2777c5401a6426c9bfc981f217417d0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a2777c5401a6426c9bfc981f217417d0 2023-04-05 09:56:14,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5f28f15108cc4d38a23ad011e88d2464 2023-04-05 09:56:14,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fe4436e7713f4987857ebc0f732de6f4 2023-04-05 09:56:14,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/60f15fbe71214421bb16044a09634adf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/60f15fbe71214421bb16044a09634adf 2023-04-05 09:56:14,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/5ffe3a088e0b4254875b7fe352ceaaf7 2023-04-05 09:56:14,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/291aa7feb909460fa145c20c1084dd7c 2023-04-05 09:56:14,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/609537646fe84038bbbf6baf9b2b16f0 2023-04-05 09:56:14,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d41c95e50d9d49c983991fdc919b791f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d41c95e50d9d49c983991fdc919b791f 2023-04-05 09:56:14,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/57b2ef6c35c7427682944c06118223be 2023-04-05 09:56:14,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f14f89572b8e45ca818d2e66db309a20 2023-04-05 09:56:14,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/b01f3b3b3d7049c892f8d224bcd7f4ac 2023-04-05 09:56:14,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f0f6ee0801d9445798c34e3389b25ecd 2023-04-05 09:56:14,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1555a508130e4b34a69efcda71ac7301 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/1555a508130e4b34a69efcda71ac7301 2023-04-05 09:56:14,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ed549f528ca444d0be25f3213c0ba88d 2023-04-05 09:56:14,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/11fd99aeb4c045f5ae25b8e1cd2779f4 2023-04-05 09:56:14,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e4c0b44f84cb43f180aa23c3ba89f54b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e4c0b44f84cb43f180aa23c3ba89f54b 2023-04-05 09:56:14,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/e1739b64db19414bb5fd52fa74a0a383 2023-04-05 09:56:14,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/d47dd2a4d3f749fb8ae9253ae3ab470a 2023-04-05 09:56:14,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/f936bf44de734ffd86be485129d7d836 2023-04-05 09:56:14,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/340dcb263f554b978f4d9a5a104c29c5 2023-04-05 09:56:14,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/be27949f36934995881e93247b9ccce0 2023-04-05 09:56:14,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4a936ad8e5d04ecf82a701ea5bdaa6f6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/b084d0de4357463ab919a18472a72ded, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f3df85725e3c485a96bc29c2cf312b9e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/a4acb7cd37b5444c90dfab5d1c00ac1a, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/615fd19830ef4120aae534f1a70504d9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/7c5506cff0bc4ced9a24f859bae90eea, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db9589f3f6ab4ca7ad29e88a74f6b6fb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f7ca7b967c2b4f848aa05e4204ca2ae0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/de5e72000f28461eb19ab356780657da, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73] to archive 2023-04-05 09:56:14,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:14,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/019b47a5bbae4dabbdb470fd7fc34f99 2023-04-05 09:56:14,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/5c82305d38c04762bb72361fba57915f 2023-04-05 09:56:14,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4a936ad8e5d04ecf82a701ea5bdaa6f6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4a936ad8e5d04ecf82a701ea5bdaa6f6 2023-04-05 09:56:14,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/c9397e082ac24e49ac5e3874c16e4df6 2023-04-05 09:56:14,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fee5ee2dcc64722b209f8ac41c78333 2023-04-05 09:56:14,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/b084d0de4357463ab919a18472a72ded to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/b084d0de4357463ab919a18472a72ded 2023-04-05 09:56:14,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/dcd292dc78124249abb93b6346b7faa7 2023-04-05 09:56:14,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ba9b142aaa5c42d0a54bee2a7b6d179e 2023-04-05 09:56:14,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/6fd72310226e44ceb68bf7f720d1d430 2023-04-05 09:56:14,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f3df85725e3c485a96bc29c2cf312b9e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f3df85725e3c485a96bc29c2cf312b9e 2023-04-05 09:56:14,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9c56baee977a47de9a6dbc86908b7bbf 2023-04-05 09:56:14,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/ff11fbcb85934af68506630884ab8ce6 2023-04-05 09:56:14,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/a4acb7cd37b5444c90dfab5d1c00ac1a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/a4acb7cd37b5444c90dfab5d1c00ac1a 2023-04-05 09:56:14,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/83e2ae03b5d24930bc8f05b97e71f1fd 2023-04-05 09:56:14,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4d8ebb2f8ed54eb0bd09c388d720d296 2023-04-05 09:56:14,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/1fd752e52da049729178d1844e7f213b 2023-04-05 09:56:14,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/42298ac2ac6e41c9967101512a1a9101 2023-04-05 09:56:14,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/615fd19830ef4120aae534f1a70504d9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/615fd19830ef4120aae534f1a70504d9 2023-04-05 09:56:14,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f6593224209b44aeb38720cc72efdfcd 2023-04-05 09:56:14,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0192155b3be7400a8149d7386ed713fc 2023-04-05 09:56:14,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/7c5506cff0bc4ced9a24f859bae90eea to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/7c5506cff0bc4ced9a24f859bae90eea 2023-04-05 09:56:14,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/15ee156c062d4dc887ba3104c9a8f6aa 2023-04-05 09:56:14,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9a4d1d84560343b39d162d3a351ce055 2023-04-05 09:56:14,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/bf629865374e4d48a959d0113065a089 2023-04-05 09:56:14,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db9589f3f6ab4ca7ad29e88a74f6b6fb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db9589f3f6ab4ca7ad29e88a74f6b6fb 2023-04-05 09:56:14,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/412e11a94411443990427620b093c98e 2023-04-05 09:56:14,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/30daf44fd6054c049988102ba8070594 2023-04-05 09:56:14,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/488435f5956d487d83c20fab14c6079f 2023-04-05 09:56:14,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9b34faa1f4a54deea5f4ecfab90cf9ac 2023-04-05 09:56:14,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f7ca7b967c2b4f848aa05e4204ca2ae0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/f7ca7b967c2b4f848aa05e4204ca2ae0 2023-04-05 09:56:14,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/492977eddbd64e29b702abbca682fab1 2023-04-05 09:56:14,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/36ea3bdef1bc4f179b6561e01256e4e0 2023-04-05 09:56:14,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/de5e72000f28461eb19ab356780657da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/de5e72000f28461eb19ab356780657da 2023-04-05 09:56:14,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/8208b9ef1bd24109a2ed82de682b5ffd 2023-04-05 09:56:14,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9bc30acab0d84b51a627e84c22dfb5f4 2023-04-05 09:56:14,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4ef6764ed1d449cebacba822bb95e30b 2023-04-05 09:56:14,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/e850b4a270c7415eab7e0e73f4f73b83 2023-04-05 09:56:14,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/4333a47e3efa4e208ef0b4530114de73 2023-04-05 09:56:14,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/recovered.edits/692.seqid, newMaxSeqId=692, maxSeqId=1 2023-04-05 09:56:14,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc. 2023-04-05 09:56:14,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 62afd6a1e0fb2ca72b38ec7d7142b5cc: 2023-04-05 09:56:14,674 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:14,674 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=62afd6a1e0fb2ca72b38ec7d7142b5cc, regionState=CLOSED 2023-04-05 09:56:14,675 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688574674"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688574674"}]},"ts":"1680688574674"} 2023-04-05 09:56:14,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-05 09:56:14,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 62afd6a1e0fb2ca72b38ec7d7142b5cc, server=jenkins-hbase4.apache.org,46827,1680688416991 in 1.4010 sec 2023-04-05 09:56:14,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-05 09:56:14,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=62afd6a1e0fb2ca72b38ec7d7142b5cc, UNASSIGN in 1.4080 sec 2023-04-05 09:56:14,678 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688574678"}]},"ts":"1680688574678"} 2023-04-05 09:56:14,679 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:56:14,681 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:56:14,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4200 sec 2023-04-05 09:56:15,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 09:56:15,371 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-05 09:56:15,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:56:15,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,374 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 09:56:15,374 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,378 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:15,380 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/recovered.edits] 2023-04-05 09:56:15,382 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1911cbaa6eaf46e4835844eb5686d421 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/1911cbaa6eaf46e4835844eb5686d421 2023-04-05 09:56:15,383 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/28d4e9a7734b40009e245ee8600429b3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/28d4e9a7734b40009e245ee8600429b3 2023-04-05 09:56:15,384 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/82dbef88ae7e482d9a15c77ef01eb030 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/82dbef88ae7e482d9a15c77ef01eb030 2023-04-05 09:56:15,385 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/96814baef27e487fb3cb7be0168e99d8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/96814baef27e487fb3cb7be0168e99d8 2023-04-05 09:56:15,386 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e0b474a7c8f64f279054c53eb743d8c1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/A/e0b474a7c8f64f279054c53eb743d8c1 2023-04-05 09:56:15,388 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4a0ab06331324b09afa216a25c57c740 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/4a0ab06331324b09afa216a25c57c740 2023-04-05 09:56:15,389 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73edc304f956409eae03259bef8e13f3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/73edc304f956409eae03259bef8e13f3 2023-04-05 09:56:15,390 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a18a3975af7648509d04e4130a6f406a to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/a18a3975af7648509d04e4130a6f406a 2023-04-05 09:56:15,391 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ace76080bbb747a2b88ecc4289cd1eb9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/ace76080bbb747a2b88ecc4289cd1eb9 2023-04-05 09:56:15,392 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fa244bb65d8043a68e415c8f0bfbb858 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/B/fa244bb65d8043a68e415c8f0bfbb858 2023-04-05 09:56:15,394 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0756b3b3200946bb9aee11643affe331 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/0756b3b3200946bb9aee11643affe331 2023-04-05 09:56:15,395 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/2bb611c4fdcb4d8291051e97641f0503 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/2bb611c4fdcb4d8291051e97641f0503 2023-04-05 09:56:15,395 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/332eb39de8c349f3911735a9f6c8f66b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/332eb39de8c349f3911735a9f6c8f66b 2023-04-05 09:56:15,396 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9dda6c10005948d8941a3bd9bea7503f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/9dda6c10005948d8941a3bd9bea7503f 2023-04-05 09:56:15,397 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db8ab576e3c04c23b80c954819ddafe8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/C/db8ab576e3c04c23b80c954819ddafe8 2023-04-05 09:56:15,400 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/recovered.edits/692.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc/recovered.edits/692.seqid 2023-04-05 09:56:15,400 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/62afd6a1e0fb2ca72b38ec7d7142b5cc 2023-04-05 09:56:15,400 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:56:15,402 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,403 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:56:15,409 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:56:15,410 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,410 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:56:15,410 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688575410"}]},"ts":"9223372036854775807"} 2023-04-05 09:56:15,411 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:56:15,411 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 62afd6a1e0fb2ca72b38ec7d7142b5cc, NAME => 'TestAcidGuarantees,,1680688542681.62afd6a1e0fb2ca72b38ec7d7142b5cc.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:56:15,412 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:56:15,412 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688575412"}]},"ts":"9223372036854775807"} 2023-04-05 09:56:15,413 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:56:15,414 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 42 msec 2023-04-05 09:56:15,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 09:56:15,475 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-05 09:56:15,487 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 228), OpenFileDescriptor=535 (was 511) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 323), ProcessCount=169 (was 169), AvailableMemoryMB=1458 (was 896) - AvailableMemoryMB LEAK? - 2023-04-05 09:56:15,498 INFO [Listener at localhost/42121] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=535, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=169, AvailableMemoryMB=1457 2023-04-05 09:56:15,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:56:15,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:56:15,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:15,502 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 09:56:15,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-05 09:56:15,503 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 09:56:15,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:15,504 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:15,504 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b empty. 2023-04-05 09:56:15,505 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:15,505 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:56:15,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:15,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:15,911 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:56:15,912 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a168ad47131a50498eee7b0d12023d0b, NAME => 'TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp 2023-04-05 09:56:16,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a168ad47131a50498eee7b0d12023d0b, disabling compactions & flushes 2023-04-05 09:56:16,317 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. after waiting 0 ms 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,317 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,317 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:16,320 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 09:56:16,321 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688576320"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688576320"}]},"ts":"1680688576320"} 2023-04-05 09:56:16,322 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 09:56:16,322 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 09:56:16,323 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688576322"}]},"ts":"1680688576322"} 2023-04-05 09:56:16,323 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 09:56:16,327 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, ASSIGN}] 2023-04-05 09:56:16,328 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, ASSIGN 2023-04-05 09:56:16,329 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=false 2023-04-05 09:56:16,480 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:16,481 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688576480"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688576480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688576480"}]},"ts":"1680688576480"} 2023-04-05 09:56:16,482 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:56:16,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:16,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a168ad47131a50498eee7b0d12023d0b, NAME => 'TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:56:16,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:56:16,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,638 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,640 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A 2023-04-05 09:56:16,640 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A 2023-04-05 09:56:16,640 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:16,640 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName A 2023-04-05 09:56:16,641 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:16,641 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,642 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B 2023-04-05 09:56:16,642 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B 2023-04-05 09:56:16,642 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:16,642 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName B 2023-04-05 09:56:16,643 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:16,643 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,644 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C 2023-04-05 09:56:16,644 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C 2023-04-05 09:56:16,644 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:16,644 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName C 2023-04-05 09:56:16,645 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:16,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:56:16,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:16,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 09:56:16,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a168ad47131a50498eee7b0d12023d0b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70959380, jitterRate=0.0573771595954895}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:56:16,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:16,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., pid=56, masterSystemTime=1680688576633 2023-04-05 09:56:16,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:16,652 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:16,652 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688576652"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688576652"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688576652"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688576652"}]},"ts":"1680688576652"} 2023-04-05 09:56:16,654 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-05 09:56:16,654 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 in 171 msec 2023-04-05 09:56:16,655 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-05 09:56:16,655 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, ASSIGN in 327 msec 2023-04-05 09:56:16,656 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 09:56:16,656 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688576656"}]},"ts":"1680688576656"} 2023-04-05 09:56:16,657 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 09:56:16,658 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 09:56:16,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-05 09:56:17,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 09:56:17,608 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-05 09:56:17,608 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x68067de1 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:17,615 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@470742aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:17,616 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:17,617 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:17,618 DEBUG [Listener at localhost/42121] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 09:56:17,620 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 09:56:17,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 09:56:17,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 09:56:17,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:18,033 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:56:18,034 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 09:56:18,034 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 09:56:18,036 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 09:56:18,038 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, REOPEN/MOVE}] 2023-04-05 09:56:18,039 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, REOPEN/MOVE 2023-04-05 09:56:18,039 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,039 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688578039"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688578039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688578039"}]},"ts":"1680688578039"} 2023-04-05 09:56:18,040 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:56:18,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a168ad47131a50498eee7b0d12023d0b, disabling compactions & flushes 2023-04-05 09:56:18,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. after waiting 0 ms 2023-04-05 09:56:18,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 09:56:18,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:18,198 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a168ad47131a50498eee7b0d12023d0b to self. 2023-04-05 09:56:18,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,199 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=CLOSED 2023-04-05 09:56:18,200 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688578199"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688578199"}]},"ts":"1680688578199"} 2023-04-05 09:56:18,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-05 09:56:18,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 in 160 msec 2023-04-05 09:56:18,202 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46827,1680688416991; forceNewPlan=false, retain=true 2023-04-05 09:56:18,353 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,353 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688578353"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688578353"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688578353"}]},"ts":"1680688578353"} 2023-04-05 09:56:18,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:56:18,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a168ad47131a50498eee7b0d12023d0b, NAME => 'TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.', STARTKEY => '', ENDKEY => ''} 2023-04-05 09:56:18,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 09:56:18,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,510 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,511 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A 2023-04-05 09:56:18,511 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A 2023-04-05 09:56:18,511 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:18,511 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName A 2023-04-05 09:56:18,513 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:18,513 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,514 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B 2023-04-05 09:56:18,514 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B 2023-04-05 09:56:18,514 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:18,514 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName B 2023-04-05 09:56:18,515 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:18,515 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,515 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C 2023-04-05 09:56:18,516 DEBUG [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C 2023-04-05 09:56:18,516 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 09:56:18,516 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a168ad47131a50498eee7b0d12023d0b columnFamilyName C 2023-04-05 09:56:18,516 INFO [StoreOpener-a168ad47131a50498eee7b0d12023d0b-1] regionserver.HStore(310): Store=a168ad47131a50498eee7b0d12023d0b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 09:56:18,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 09:56:18,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a168ad47131a50498eee7b0d12023d0b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59643579, jitterRate=-0.111241415143013}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 09:56:18,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:18,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., pid=61, masterSystemTime=1680688578505 2023-04-05 09:56:18,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,525 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,525 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688578525"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680688578525"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680688578525"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680688578525"}]},"ts":"1680688578525"} 2023-04-05 09:56:18,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-05 09:56:18,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 in 172 msec 2023-04-05 09:56:18,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-05 09:56:18,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, REOPEN/MOVE in 489 msec 2023-04-05 09:56:18,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-05 09:56:18,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-05 09:56:18,531 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-05 09:56:18,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-05 09:56:18,532 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,539 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,539 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,543 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,544 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,548 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,548 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,552 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,552 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,560 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,560 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,566 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,566 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,570 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,570 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,577 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,577 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,582 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,582 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:62753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 09:56:18,590 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 09:56:18,601 DEBUG [hconnection-0x6d97b28a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,601 DEBUG [hconnection-0x544cf12e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,601 DEBUG [hconnection-0x64f804d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,602 DEBUG [hconnection-0x319e45c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,603 DEBUG [hconnection-0x322b7d2c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,604 DEBUG [hconnection-0x482af8a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,604 DEBUG [hconnection-0x4e6921ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,605 DEBUG [hconnection-0x3cf52982-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,606 DEBUG [hconnection-0x45e329a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,606 DEBUG [hconnection-0x4f55c6c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 09:56:18,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,607 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,610 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,610 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,610 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,610 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,610 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:18,616 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 09:56:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:18,621 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:18,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:18,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:18,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,622 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,623 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,623 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,624 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,624 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:18,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:18,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:18,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:18,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:18,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:18,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:18,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:18,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:18,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:18,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:18,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:18,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:18,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:18,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:18,634 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,635 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f01f21[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:18,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:18,637 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,637 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:18,639 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:18,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,640 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:18,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:18,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,641 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,641 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:18,644 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:18,644 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:18,644 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:18,644 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:18,644 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:18,646 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,646 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,646 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,646 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,647 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,649 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:18,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:18,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,650 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,650 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:18,650 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:18,657 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058f3cec1b8d6f4954a82194e36a2ea439 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f3cec1b8d6f4954a82194e36a2ea439 2023-04-05 09:56:18,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688638662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688638662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688638663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688638664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688638664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,750 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:18,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:18,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:18,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:18,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:18,754 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:18,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:18,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:18,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,755 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,755 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,756 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:18,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:18,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:18,758 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,758 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:18,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bbede34[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:18,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:18,760 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,762 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:18,762 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,762 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:18,763 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,763 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688638764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688638764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:18,766 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:18,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:18,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:18,767 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:18,767 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:18,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688638768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688638768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688638768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,770 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,771 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:18,773 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:18,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,773 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,774 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,774 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:18,774 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:18,874 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:18,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:18,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:18,877 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:18,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:18,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:18,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,878 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,884 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,884 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,884 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,884 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:18,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:18,886 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,886 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:18,886 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:18,886 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:18,887 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,887 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3089d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:18,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:18,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,890 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:18,890 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,890 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:18,891 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:18,893 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:18,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:18,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:18,894 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:18,894 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:18,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:18,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,897 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,898 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:18,904 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:18,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:18,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:18,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:18,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:18,905 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:18,905 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:18,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688638967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688638968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688638970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688638971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:18,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:18,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688638972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,005 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:19,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:19,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:19,009 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:19,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:19,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:19,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:19,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,009 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,011 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:19,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,011 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:19,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:19,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:19,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:19,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:19,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:19,014 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:19,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:19,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:19,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:19,015 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:19,018 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:19,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:19,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9ea3b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:19,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:19,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:19,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:19,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:19,025 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:19,025 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:19,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:19,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:19,026 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:19,027 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:19,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:19,028 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:19,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:19,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:19,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:19,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:19,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:19,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:19,033 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:19,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:19,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:19,033 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:19,033 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:19,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,067 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7163df12b5c8427abd1a3c65f2c8b074 2023-04-05 09:56:19,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/17d18f418f474ae288bda9e7abc3f15f 2023-04-05 09:56:19,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/01c12a8b34e34aafaad62a5380ae8f6b 2023-04-05 09:56:19,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7163df12b5c8427abd1a3c65f2c8b074 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074 2023-04-05 09:56:19,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074, entries=150, sequenceid=17, filesize=25.1 K 2023-04-05 09:56:19,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/17d18f418f474ae288bda9e7abc3f15f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f 2023-04-05 09:56:19,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 09:56:19,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/01c12a8b34e34aafaad62a5380ae8f6b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b 2023-04-05 09:56:19,133 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:19,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 09:56:19,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a168ad47131a50498eee7b0d12023d0b in 506ms, sequenceid=17, compaction requested=false 2023-04-05 09:56:19,139 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 09:56:19,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:19,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:19,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:19,164 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:19,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:19,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:19,164 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:19,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,165 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,166 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:19,166 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,167 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:19,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:19,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:19,184 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:19,184 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:19,184 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:19,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:19,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688639298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688639299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688639299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688639300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688639301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688639402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688639402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688639402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688639403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688639404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,605 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054f8770eb7c3546aa9b92649380958cec to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054f8770eb7c3546aa9b92649380958cec 2023-04-05 09:56:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688639606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688639606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688639606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688639606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688639607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688639909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688639909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688639910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688639911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688639911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,010 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/e6bed63474ef438c995230d1a46eae95 2023-04-05 09:56:20,195 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 09:56:20,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:20,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688640413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:20,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688640414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:20,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688640415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:20,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688640415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:20,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688640417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,420 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/818eeb38a9364ffaa6fd018fe937daa4 2023-04-05 09:56:20,459 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/d38fdc088b9e470aba79963fd6b2b9c1 2023-04-05 09:56:20,464 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/e6bed63474ef438c995230d1a46eae95 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95 2023-04-05 09:56:20,468 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95, entries=150, sequenceid=36, filesize=25.1 K 2023-04-05 09:56:20,469 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/818eeb38a9364ffaa6fd018fe937daa4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4 2023-04-05 09:56:20,473 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4, entries=150, sequenceid=36, filesize=11.6 K 2023-04-05 09:56:20,474 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/d38fdc088b9e470aba79963fd6b2b9c1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1 2023-04-05 09:56:20,479 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1, entries=150, sequenceid=36, filesize=11.6 K 2023-04-05 09:56:20,480 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a168ad47131a50498eee7b0d12023d0b in 1296ms, sequenceid=36, compaction requested=false 2023-04-05 09:56:20,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:20,480 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,482 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2991fb14[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:20,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:20,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,486 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:20,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,486 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:20,487 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:20,493 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:20,493 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:20,493 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:20,493 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:20,493 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:20,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:20,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:20,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:20,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:20,499 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:20,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:20,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:20,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:20,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:20,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:20,503 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:20,503 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:20,502 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:20,502 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:20,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:20,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:20,503 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:20,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:20,603 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:20,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:20,606 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:20,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:20,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:20,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:20,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:20,608 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:20,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:20,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:20,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:20,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:20,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:20,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:20,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:20,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:20,620 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:20,620 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:20,621 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:20,649 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405dcf0b0bd6a5b4fc7b934164fb5637708 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dcf0b0bd6a5b4fc7b934164fb5637708 2023-04-05 09:56:21,058 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/9878d078fd1e4efc9a93f39a3bb465b7 2023-04-05 09:56:21,087 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/a7e60f624576458d81d8974410db5ff5 2023-04-05 09:56:21,110 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/5994c1bc97944c5a83ca20d7c1df4e79 2023-04-05 09:56:21,114 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/9878d078fd1e4efc9a93f39a3bb465b7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7 2023-04-05 09:56:21,117 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7, entries=150, sequenceid=49, filesize=25.1 K 2023-04-05 09:56:21,117 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/a7e60f624576458d81d8974410db5ff5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5 2023-04-05 09:56:21,121 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 09:56:21,121 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/5994c1bc97944c5a83ca20d7c1df4e79 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79 2023-04-05 09:56:21,125 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 09:56:21,126 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for a168ad47131a50498eee7b0d12023d0b in 506ms, sequenceid=49, compaction requested=true 2023-04-05 09:56:21,126 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:21,126 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,129 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7483cf6c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,131 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,132 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,135 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,138 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,138 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,138 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,140 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,140 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,140 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,142 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,145 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,145 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,145 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,145 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,145 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,245 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,256 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,256 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,257 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,258 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:21,260 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,265 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a89681[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,269 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,270 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,271 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,271 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,273 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,273 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,277 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,280 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,280 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,280 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,380 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,384 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,388 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,388 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,393 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f07219d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,395 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,395 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,396 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,396 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,396 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,399 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,402 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,402 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,402 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,402 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,402 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,403 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,403 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,403 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,404 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,406 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,408 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,408 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,409 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,409 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,409 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:21,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:21,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:21,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688641489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688641491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688641492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,497 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040517d3572bf9ca4572974e6b6ca784e0b7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040517d3572bf9ca4572974e6b6ca784e0b7 2023-04-05 09:56:21,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688641493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688641494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,509 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,520 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,521 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,524 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,525 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,525 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:21,544 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,546 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e82832[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,548 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,549 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,550 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,552 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,552 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,554 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,554 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,554 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,554 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,554 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,556 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,556 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,556 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,557 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,558 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,560 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,561 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,561 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,561 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,561 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688641595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688641596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688641596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688641598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688641598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,661 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,665 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,665 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,667 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,667 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,668 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,669 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,669 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:21,670 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,671 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,671 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,672 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137a70cc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,674 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,678 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,678 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,679 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,680 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,682 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,682 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,682 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,684 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,686 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,689 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,689 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,689 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,689 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,789 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,796 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,796 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,797 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,797 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,799 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,799 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688641797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688641798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,801 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688641799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688641801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:21,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688641803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,809 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fcaf0bc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,811 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,813 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,814 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,815 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,818 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,818 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,820 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,821 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,823 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,825 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,825 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,826 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,826 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/ddfe4cdb6c424a3382da2ea7d39ae1b2 2023-04-05 09:56:21,926 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:21,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:21,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:21,930 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:21,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:21,930 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:21,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,931 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,932 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/c79f6ddb39244ffab0784a370d16039d 2023-04-05 09:56:21,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,933 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:21,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:21,936 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,936 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:21,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:21,937 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:21,938 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59272511[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:21,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:21,943 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,945 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:21,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:21,947 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:21,954 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:21,954 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:21,954 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:21,954 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:21,954 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:21,956 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,956 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,956 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:21,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:21,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:21,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/15bc53d994b74771865d16d56bdcc550 2023-04-05 09:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,958 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:21,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:21,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:21,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:21,962 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:21,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:21,962 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:21,962 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:21,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/ddfe4cdb6c424a3382da2ea7d39ae1b2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2 2023-04-05 09:56:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:21,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2, entries=150, sequenceid=63, filesize=25.1 K 2023-04-05 09:56:21,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/c79f6ddb39244ffab0784a370d16039d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d 2023-04-05 09:56:21,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d, entries=150, sequenceid=63, filesize=11.6 K 2023-04-05 09:56:21,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/15bc53d994b74771865d16d56bdcc550 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550 2023-04-05 09:56:21,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550, entries=150, sequenceid=63, filesize=11.6 K 2023-04-05 09:56:21,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a168ad47131a50498eee7b0d12023d0b in 500ms, sequenceid=63, compaction requested=true 2023-04-05 09:56:21,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:21,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:21,974 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:21,974 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:21,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:21,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/A is initiating minor compaction (all files) 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:21,975 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/A in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,975 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:21,975 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=100.6 K 2023-04-05 09:56:21,975 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=46.4 K 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 7163df12b5c8427abd1a3c65f2c8b074, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688578624 2023-04-05 09:56:21,975 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 17d18f418f474ae288bda9e7abc3f15f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688578624 2023-04-05 09:56:21,976 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e6bed63474ef438c995230d1a46eae95, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680688578653 2023-04-05 09:56:21,976 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 9878d078fd1e4efc9a93f39a3bb465b7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688579298 2023-04-05 09:56:21,976 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 818eeb38a9364ffaa6fd018fe937daa4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680688578653 2023-04-05 09:56:21,977 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ddfe4cdb6c424a3382da2ea7d39ae1b2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:21,977 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a7e60f624576458d81d8974410db5ff5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688579298 2023-04-05 09:56:21,977 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c79f6ddb39244ffab0784a370d16039d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:21,985 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#502 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:21,987 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#A#compaction#501 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:22,005 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/9656d12cebc14cdf94633bb9f48511cd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/9656d12cebc14cdf94633bb9f48511cd 2023-04-05 09:56:22,009 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into 9656d12cebc14cdf94633bb9f48511cd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:22,009 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:22,009 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=12, startTime=1680688581974; duration=0sec 2023-04-05 09:56:22,009 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:22,010 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:22,011 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:22,011 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:22,011 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:22,011 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=46.4 K 2023-04-05 09:56:22,011 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 01c12a8b34e34aafaad62a5380ae8f6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680688578624 2023-04-05 09:56:22,012 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d38fdc088b9e470aba79963fd6b2b9c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680688578653 2023-04-05 09:56:22,012 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5994c1bc97944c5a83ca20d7c1df4e79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680688579298 2023-04-05 09:56:22,012 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 15bc53d994b74771865d16d56bdcc550, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:22,018 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:22,062 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:22,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:22,066 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:22,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:22,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:22,066 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:22,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:22,068 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:22,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:22,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:22,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:22,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:22,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:22,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:22,069 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:22,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:22,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:22,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:22,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:22,070 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:22,070 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:22,094 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405bb65a410b9144d21b71795b5c1abc248 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bb65a410b9144d21b71795b5c1abc248 2023-04-05 09:56:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:22,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688642110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688642110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688642115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688642115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688642115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688642216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688642220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688642220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688642220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688642419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688642422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,425 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/8ac2b3fffbc64b86bb0bf64621a9f0bf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8ac2b3fffbc64b86bb0bf64621a9f0bf 2023-04-05 09:56:22,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688642424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688642424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,429 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into 8ac2b3fffbc64b86bb0bf64621a9f0bf(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:22,429 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:22,429 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=12, startTime=1680688581974; duration=0sec 2023-04-05 09:56:22,429 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:22,499 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/2b0e87f991774118ac9a640fa39821f9 2023-04-05 09:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688642618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688642722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688642725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688642726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:22,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688642727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,820 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/519cf4af787144978145d6c1482e1660 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/519cf4af787144978145d6c1482e1660 2023-04-05 09:56:22,824 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/A of a168ad47131a50498eee7b0d12023d0b into 519cf4af787144978145d6c1482e1660(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:22,824 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:22,824 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/A, priority=12, startTime=1680688581974; duration=0sec 2023-04-05 09:56:22,824 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:22,911 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/c15970f2b58e47f5ab3d5a25eba243b0 2023-04-05 09:56:22,946 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4d805c957fd04e47a104c0fb67a65b99 2023-04-05 09:56:22,950 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/2b0e87f991774118ac9a640fa39821f9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9 2023-04-05 09:56:22,954 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9, entries=150, sequenceid=84, filesize=25.1 K 2023-04-05 09:56:22,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/c15970f2b58e47f5ab3d5a25eba243b0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0 2023-04-05 09:56:22,962 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 09:56:22,962 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4d805c957fd04e47a104c0fb67a65b99 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99 2023-04-05 09:56:22,966 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 09:56:22,967 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a168ad47131a50498eee7b0d12023d0b in 897ms, sequenceid=84, compaction requested=false 2023-04-05 09:56:22,967 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:22,967 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:22,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:22,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:22,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:22,967 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:22,969 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b63fbf[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:22,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:22,972 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,975 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:22,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:22,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:22,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,976 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:22,978 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:22,981 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:22,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:22,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:22,981 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:22,981 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:22,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,987 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:22,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:22,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:22,996 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:22,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:22,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:22,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:22,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:22,997 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:22,997 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:23,097 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:23,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:23,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:23,104 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:23,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:23,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:23,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:23,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:23,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:23,106 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:23,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:23,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:23,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:23,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:23,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:23,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:23,107 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:23,108 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:23,108 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:23,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:23,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688643242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688643242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688643244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688643245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688643346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688643346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688643348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688643348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,533 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405008ce3b678ab47f2be2a8c6c8c8cc1be to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405008ce3b678ab47f2be2a8c6c8c8cc1be 2023-04-05 09:56:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688643548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688643550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688643550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688643550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688643628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688643852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688643853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688643854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688643854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:23,942 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b 2023-04-05 09:56:24,358 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3da2a906a45a42a295d0682f084d4853 2023-04-05 09:56:24,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:24,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688644357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688644357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688644358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:24,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688644359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,390 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/47dcdc6b3f9b481584bcde5c883677f4 2023-04-05 09:56:24,397 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b 2023-04-05 09:56:24,401 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b, entries=150, sequenceid=98, filesize=25.1 K 2023-04-05 09:56:24,402 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3da2a906a45a42a295d0682f084d4853 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853 2023-04-05 09:56:24,417 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853, entries=150, sequenceid=98, filesize=11.6 K 2023-04-05 09:56:24,418 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/47dcdc6b3f9b481584bcde5c883677f4 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4 2023-04-05 09:56:24,424 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4, entries=150, sequenceid=98, filesize=11.6 K 2023-04-05 09:56:24,425 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a168ad47131a50498eee7b0d12023d0b in 1317ms, sequenceid=98, compaction requested=true 2023-04-05 09:56:24,425 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:24,425 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:24,427 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@218adf55[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:24,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:24,438 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,439 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,439 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:24,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:24,442 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:24,447 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:24,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:24,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:24,447 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:24,447 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:24,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,449 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:24,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,449 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,450 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:24,453 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:24,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:24,453 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:24,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:24,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:24,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:24,453 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:24,454 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:24,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,454 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:24,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:24,554 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:24,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:24,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:24,583 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:24,583 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:24,583 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:24,583 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:24,584 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,584 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:24,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,585 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:24,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:24,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:24,585 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:24,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:24,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:24,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:24,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:24,612 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:24,612 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:24,612 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:24,612 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:24,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:24,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:24,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:24,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:24,613 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:24,655 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058ae644903a204213b49af580f9b0733f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058ae644903a204213b49af580f9b0733f 2023-04-05 09:56:24,666 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/29e224429dec47b2aa301736992a00ae 2023-04-05 09:56:25,082 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/4004d924ddb34caca322d3912948e5a5 2023-04-05 09:56:25,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:25,364 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:25,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688645372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688645373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688645373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688645374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688645483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688645483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688645483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688645483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,493 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/6fe1d88457f94098b05ed966f1084fb5 2023-04-05 09:56:25,497 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/29e224429dec47b2aa301736992a00ae as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae 2023-04-05 09:56:25,500 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae, entries=150, sequenceid=118, filesize=25.1 K 2023-04-05 09:56:25,501 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/4004d924ddb34caca322d3912948e5a5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5 2023-04-05 09:56:25,503 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5, entries=150, sequenceid=118, filesize=11.6 K 2023-04-05 09:56:25,504 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/6fe1d88457f94098b05ed966f1084fb5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5 2023-04-05 09:56:25,507 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5, entries=150, sequenceid=118, filesize=11.6 K 2023-04-05 09:56:25,508 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a168ad47131a50498eee7b0d12023d0b in 896ms, sequenceid=118, compaction requested=true 2023-04-05 09:56:25,508 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:25,508 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:25,509 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492d94c8[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:25,511 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:25,511 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,513 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:25,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:25,513 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:25,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,513 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:25,515 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:25,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:25,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:25,515 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:25,517 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:25,517 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:25,517 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:25,517 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:25,517 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:25,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,519 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:25,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:25,519 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,520 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,522 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:25,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:25,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:25,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:25,523 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:25,523 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:25,523 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:25,624 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:25,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:25,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:25,628 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:25,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:25,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:25,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:25,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,628 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:25,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:25,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:25,630 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:25,630 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,630 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:25,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:25,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:25,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:25,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:25,631 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:25,631 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:25,631 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:25,632 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:25,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:25,641 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:25,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688645687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688645690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688645691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688645692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688645692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688645795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688645795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688645797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688645798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:25,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:25,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688645995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688646001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688646001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688646001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688646001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,066 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040598930516432d4272804dd3687e9432da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040598930516432d4272804dd3687e9432da 2023-04-05 09:56:26,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688646304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688646305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688646305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688646306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,472 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/22af08849628412e99816a0d8dce6504 2023-04-05 09:56:26,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688646498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688646810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688646811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688646812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688646813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:26,895 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/f794231a20ef47eb88c4fe49987db981 2023-04-05 09:56:27,324 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/31764b1c1f25481da803f5784ae4a065 2023-04-05 09:56:27,327 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/22af08849628412e99816a0d8dce6504 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504 2023-04-05 09:56:27,330 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504, entries=150, sequenceid=130, filesize=25.2 K 2023-04-05 09:56:27,331 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/f794231a20ef47eb88c4fe49987db981 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981 2023-04-05 09:56:27,334 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981, entries=150, sequenceid=130, filesize=11.7 K 2023-04-05 09:56:27,335 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/31764b1c1f25481da803f5784ae4a065 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065 2023-04-05 09:56:27,343 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065, entries=150, sequenceid=130, filesize=11.7 K 2023-04-05 09:56:27,344 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a168ad47131a50498eee7b0d12023d0b in 1713ms, sequenceid=130, compaction requested=true 2023-04-05 09:56:27,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:27,344 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,344 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:27,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:27,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:27,345 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:27,346 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:27,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29423f51[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:27,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:27,348 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,349 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:27,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:27,350 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:27,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:27,351 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:27,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:27,356 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:27,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:27,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:27,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:27,356 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:27,359 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,359 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,359 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,360 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,362 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,371 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,371 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,372 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:27,372 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:27,372 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:27,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:27,472 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:27,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:27,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:27,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:27,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:27,477 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:27,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:27,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:27,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,478 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,479 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,479 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,480 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:27,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:27,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:27,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:27,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,482 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:27,484 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:27,484 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:27,484 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,484 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:27,485 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:27,485 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:27,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:27,509 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304059ae4055b655e4e0388784a5a90603f1d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059ae4055b655e4e0388784a5a90603f1d 2023-04-05 09:56:27,524 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/e69ff35341484ceb95b74e5c62afcf36 2023-04-05 09:56:27,544 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/99bb83634f8b4c0f885e386c9d735212 2023-04-05 09:56:27,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688647548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,557 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/5c58bbbe663b45b08ce620bfb4d0615f 2023-04-05 09:56:27,562 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/e69ff35341484ceb95b74e5c62afcf36 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36 2023-04-05 09:56:27,567 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36, entries=150, sequenceid=150, filesize=25.3 K 2023-04-05 09:56:27,567 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/99bb83634f8b4c0f885e386c9d735212 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212 2023-04-05 09:56:27,571 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212, entries=150, sequenceid=150, filesize=11.8 K 2023-04-05 09:56:27,572 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/5c58bbbe663b45b08ce620bfb4d0615f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f 2023-04-05 09:56:27,575 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f, entries=150, sequenceid=150, filesize=11.8 K 2023-04-05 09:56:27,576 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a168ad47131a50498eee7b0d12023d0b in 91ms, sequenceid=150, compaction requested=true 2023-04-05 09:56:27,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:27,576 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:27,578 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d9b601[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:27,580 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:27,581 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,581 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:27,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,582 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:27,583 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:27,586 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:27,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:27,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:27,586 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:27,586 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:27,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,587 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,588 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,588 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,589 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:27,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,593 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,592 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,593 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:27,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,593 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:27,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:27,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:27,693 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:27,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:27,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:27,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:27,697 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,698 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,698 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:27,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:27,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:27,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:27,700 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:27,701 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e5eae18[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:27,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:27,703 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,704 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:27,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,704 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:27,706 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:27,708 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:27,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:27,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:27,708 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:27,708 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:27,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,710 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,711 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:27,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:27,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:27,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,713 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,714 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,714 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:27,714 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:27,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688647733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,814 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:27,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:27,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:27,817 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:27,817 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,817 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:27,817 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:27,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,818 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,819 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688647815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,819 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688647816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:27,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:27,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:27,821 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,821 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:27,821 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:27,821 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:27,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688647822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688647824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,826 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf0f0cf[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:27,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:27,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,829 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:27,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:27,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:27,832 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:27,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:27,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:27,833 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:27,833 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:27,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,834 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688647837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:27,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,841 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,841 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,841 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:27,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:27,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,842 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:27,942 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:27,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:27,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:27,945 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,946 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,946 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:27,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:27,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:27,948 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,948 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:27,948 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:27,948 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:27,950 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:27,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1370fa57[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:27,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:27,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:27,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:27,955 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:27,957 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:27,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:27,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:27,957 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:27,957 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:27,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,959 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:27,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:27,960 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,961 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:27,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:27,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:27,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:27,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:27,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:27,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:27,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:27,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:27,964 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:27,964 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:27,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688648040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,064 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,067 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,067 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,069 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,069 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,069 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,069 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,071 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,071 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,071 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,071 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,073 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272d2f86[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,075 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,076 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,076 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,078 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,078 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,081 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,081 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,081 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,082 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304050e56fe5bc16c4202a2512ae9f3adcc68 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050e56fe5bc16c4202a2512ae9f3adcc68 2023-04-05 09:56:28,084 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,084 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,084 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,084 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,085 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,090 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,091 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,091 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,091 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,191 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,196 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,196 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,198 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,198 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,199 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,199 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,200 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,203 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf86b15[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,205 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,206 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,206 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,206 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,209 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,210 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,214 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,214 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,216 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,217 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,220 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,220 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,221 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,221 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,221 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:28,321 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,325 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,326 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,326 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,327 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,327 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,329 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,334 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70228ecd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,336 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,337 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,338 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,338 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,339 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:28,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688648343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,345 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,345 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,345 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,348 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,349 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,350 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,353 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,353 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,354 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,354 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,454 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,458 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,458 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,459 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,459 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,463 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,464 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,465 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,472 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,472 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,472 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,472 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,474 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641460dc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,477 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,477 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,480 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,482 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,482 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,482 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,483 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,483 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,483 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,485 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,487 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/6f7a18fcffb447b994fd5da89caa7b35 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,488 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,489 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,489 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,489 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,489 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,489 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,589 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,592 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,592 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,593 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,594 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,594 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,599 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,599 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0b32c7[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,602 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,603 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,606 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,606 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,606 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,608 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,608 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,608 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,608 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,610 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,613 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,613 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,613 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,613 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,614 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,714 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,717 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,717 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,721 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,725 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,725 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,725 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,725 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,726 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,726 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,727 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,727 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,728 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b44450[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,729 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,737 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,737 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,738 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,741 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,742 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,743 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,744 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,745 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,746 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,747 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,747 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,747 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,847 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,851 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,851 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,851 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,851 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,851 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,851 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:28,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688648849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,853 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,854 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,855 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,855 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,855 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,855 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,855 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5184e0cd[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,859 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,860 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,860 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,861 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,861 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,863 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,863 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,863 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,866 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,866 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,867 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,870 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,870 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,870 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/8bda470701b040f19e67423508a432e9 2023-04-05 09:56:28,970 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:28,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:28,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:28,973 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:28,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:28,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:28,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,974 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,975 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,975 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,975 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,975 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:28,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:28,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:28,977 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,977 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:28,977 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:28,977 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,978 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,979 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d168a1[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:28,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:28,980 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,982 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:28,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,983 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,985 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:28,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:28,991 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:28,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:28,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:28,991 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:28,991 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:28,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:28,994 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,995 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:28,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:28,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:28,998 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:28,998 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:28,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:28,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,000 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,099 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:29,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:29,101 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:29,101 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:29,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:29,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,104 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:29,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:29,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:29,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:29,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:29,105 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:29,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bfedb6[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:29,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:29,108 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,109 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:29,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,110 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:29,110 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:29,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,110 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,111 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:29,113 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:29,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:29,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:29,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:29,113 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:29,117 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,117 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,117 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,118 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,122 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,123 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,124 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:29,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:29,124 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,124 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:29,124 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:29,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,128 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,224 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:29,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:29,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:29,227 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:29,227 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,229 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,229 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:29,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:29,231 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,231 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:29,231 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:29,231 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,233 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723eb395[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:29,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:29,235 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,237 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,237 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:29,237 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:29,238 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:29,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,238 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:29,239 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:29,242 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:29,243 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:29,243 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:29,243 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:29,243 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:29,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,244 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,245 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,247 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,251 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,251 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,251 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,251 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:29,252 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,251 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,252 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,252 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,252 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:29,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,252 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,253 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:29,253 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:29,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/218fdca633d14dc086bb1e789d319ca1 2023-04-05 09:56:29,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/6f7a18fcffb447b994fd5da89caa7b35 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35 2023-04-05 09:56:29,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35, entries=200, sequenceid=163, filesize=32.1 K 2023-04-05 09:56:29,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/8bda470701b040f19e67423508a432e9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9 2023-04-05 09:56:29,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9, entries=150, sequenceid=163, filesize=11.8 K 2023-04-05 09:56:29,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/218fdca633d14dc086bb1e789d319ca1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1 2023-04-05 09:56:29,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1, entries=150, sequenceid=163, filesize=11.8 K 2023-04-05 09:56:29,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a168ad47131a50498eee7b0d12023d0b in 1662ms, sequenceid=163, compaction requested=true 2023-04-05 09:56:29,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:29,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:29,321 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:56:29,321 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:56:29,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:29,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:29,323 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 187700 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:56:29,323 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/A is initiating minor compaction (all files) 2023-04-05 09:56:29,323 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83690 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:56:29,323 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/A in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,323 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:29,323 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/519cf4af787144978145d6c1482e1660, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=183.3 K 2023-04-05 09:56:29,323 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,324 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/9656d12cebc14cdf94633bb9f48511cd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=81.7 K 2023-04-05 09:56:29,324 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 519cf4af787144978145d6c1482e1660, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:29,324 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 9656d12cebc14cdf94633bb9f48511cd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:29,324 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2b0e87f991774118ac9a640fa39821f9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688581488 2023-04-05 09:56:29,324 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting c15970f2b58e47f5ab3d5a25eba243b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688581488 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c85e90a2ca4044ca8f0b7e2d5b3faf9b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680688582107 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 3da2a906a45a42a295d0682f084d4853, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680688582107 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4004d924ddb34caca322d3912948e5a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680688583240 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 29e224429dec47b2aa301736992a00ae, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680688583240 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f794231a20ef47eb88c4fe49987db981, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680688585372 2023-04-05 09:56:29,325 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 22af08849628412e99816a0d8dce6504, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680688585372 2023-04-05 09:56:29,326 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 99bb83634f8b4c0f885e386c9d735212, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680688585689 2023-04-05 09:56:29,326 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting e69ff35341484ceb95b74e5c62afcf36, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680688585689 2023-04-05 09:56:29,326 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8bda470701b040f19e67423508a432e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:29,326 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6f7a18fcffb447b994fd5da89caa7b35, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:29,343 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#522 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:29,349 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#A#compaction#523 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:29,352 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/f5d15092c39e42319d6838595ecff981 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f5d15092c39e42319d6838595ecff981 2023-04-05 09:56:29,353 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:29,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:29,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:29,356 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into f5d15092c39e42319d6838595ecff981(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:29,356 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:29,356 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=9, startTime=1680688589321; duration=0sec 2023-04-05 09:56:29,356 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:29,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,356 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:29,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:29,356 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:29,356 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:56:29,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,359 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,359 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,359 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,359 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:29,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:29,361 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:29,361 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,364 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83690 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:56:29,364 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:29,364 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,364 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8ac2b3fffbc64b86bb0bf64621a9f0bf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=81.7 K 2023-04-05 09:56:29,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8ac2b3fffbc64b86bb0bf64621a9f0bf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680688581438 2023-04-05 09:56:29,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4d805c957fd04e47a104c0fb67a65b99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680688581488 2023-04-05 09:56:29,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 47dcdc6b3f9b481584bcde5c883677f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680688582107 2023-04-05 09:56:29,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6fe1d88457f94098b05ed966f1084fb5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680688583240 2023-04-05 09:56:29,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 31764b1c1f25481da803f5784ae4a065, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680688585372 2023-04-05 09:56:29,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 5c58bbbe663b45b08ce620bfb4d0615f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680688585689 2023-04-05 09:56:29,366 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 218fdca633d14dc086bb1e789d319ca1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:29,379 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#525 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:29,382 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054a93a9c83c044a1eb7439fb8ee9c3452 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054a93a9c83c044a1eb7439fb8ee9c3452 2023-04-05 09:56:29,793 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/54e57d89de8640739ad99b93250dfcdf as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/54e57d89de8640739ad99b93250dfcdf 2023-04-05 09:56:29,798 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/fe847d9bb5ba4f20a8194250b2d7b62d 2023-04-05 09:56:29,799 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into 54e57d89de8640739ad99b93250dfcdf(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:29,799 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:29,799 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=9, startTime=1680688589322; duration=0sec 2023-04-05 09:56:29,799 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:29,809 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3ad48ecdb83642caa0a62c1c4d69fba9 2023-04-05 09:56:29,827 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/6fd2fbebedbb4b5f970ab86b4048bcba 2023-04-05 09:56:29,831 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/fe847d9bb5ba4f20a8194250b2d7b62d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d 2023-04-05 09:56:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:29,833 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:29,835 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d, entries=150, sequenceid=183, filesize=25.3 K 2023-04-05 09:56:29,836 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3ad48ecdb83642caa0a62c1c4d69fba9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9 2023-04-05 09:56:29,841 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9, entries=150, sequenceid=183, filesize=11.8 K 2023-04-05 09:56:29,842 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/6fd2fbebedbb4b5f970ab86b4048bcba as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba 2023-04-05 09:56:29,849 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba, entries=150, sequenceid=183, filesize=11.8 K 2023-04-05 09:56:29,850 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for a168ad47131a50498eee7b0d12023d0b in 489ms, sequenceid=183, compaction requested=false 2023-04-05 09:56:29,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:29,850 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:29,852 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:29,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ab797[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:29,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:29,855 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,857 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:29,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,857 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:29,859 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:29,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,860 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:29,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:29,865 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:29,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:29,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:29,865 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:29,865 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:29,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,868 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,870 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,875 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:29,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:29,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:29,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:29,876 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:29,876 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:29,894 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040580c3e6748d174e6d9f57210ccbf63dc0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580c3e6748d174e6d9f57210ccbf63dc0 2023-04-05 09:56:29,902 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/4de6869013604bdf8e2ba8de21b126dd 2023-04-05 09:56:29,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:29,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688649900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:29,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688649901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:29,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688589891, max timestamp=1680688589907Num uniques 859; 2023-04-05 09:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:56:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688649902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688589891, max timestamp=1680688589907Num uniques 859; 2023-04-05 09:56:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:56:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680688589891, max timestamp=1680688589907Num uniques 859; 2023-04-05 09:56:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 09:56:29,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688649941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688649943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,977 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:29,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:29,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:29,986 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:29,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:29,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:29,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,987 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,989 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:29,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:29,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,989 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:29,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:29,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:29,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:29,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:29,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:29,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:29,992 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,993 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c353df[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:29,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:29,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,998 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:29,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:29,998 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:29,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:30,002 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:30,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:30,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:30,003 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:30,003 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:30,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,004 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688650009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688650010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688650010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,014 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:30,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,021 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,022 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:30,022 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688650048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688650049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,122 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:30,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:30,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:30,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:30,127 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:30,127 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,127 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:30,127 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:30,128 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,128 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,129 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,129 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:30,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:30,132 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,132 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:30,132 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:30,132 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:30,133 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,133 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38bcf0c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:30,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:30,135 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,136 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:30,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,136 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:30,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:30,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:30,141 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:30,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:30,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:30,141 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:30,141 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:30,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,143 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,144 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,146 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:30,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:30,146 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:30,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,147 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:30,147 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:30,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688650210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688650211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688650212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,247 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:30,252 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,253 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,254 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,254 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688650250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:30,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688650254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:30,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:30,262 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,262 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:30,262 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:30,262 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:30,265 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae76f3[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:30,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:30,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:30,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:30,270 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:30,273 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:30,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:30,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:30,273 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:30,273 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:30,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,275 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:30,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,279 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:30,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,280 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:30,280 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,280 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,280 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,280 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:30,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/0d0fc1c7b6954f17818cba6203b5debc 2023-04-05 09:56:30,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/cf1393fe89964aa1ae753be75f87d869 2023-04-05 09:56:30,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/4de6869013604bdf8e2ba8de21b126dd as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd 2023-04-05 09:56:30,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd, entries=350, sequenceid=195, filesize=52.6 K 2023-04-05 09:56:30,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/0d0fc1c7b6954f17818cba6203b5debc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc 2023-04-05 09:56:30,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc, entries=150, sequenceid=195, filesize=11.8 K 2023-04-05 09:56:30,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/cf1393fe89964aa1ae753be75f87d869 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869 2023-04-05 09:56:30,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869, entries=150, sequenceid=195, filesize=11.8 K 2023-04-05 09:56:30,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a168ad47131a50498eee7b0d12023d0b in 484ms, sequenceid=195, compaction requested=true 2023-04-05 09:56:30,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:30,356 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:30,356 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:30,356 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 09:56:30,356 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 7 compacting, 2 eligible, 16 blocking 2023-04-05 09:56:30,357 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 09:56:30,357 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 09:56:30,357 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. because compaction request was cancelled 2023-04-05 09:56:30,357 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:30,358 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:30,358 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:30,358 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,358 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f5d15092c39e42319d6838595ecff981, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=35.6 K 2023-04-05 09:56:30,358 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f5d15092c39e42319d6838595ecff981, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:30,358 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 3ad48ecdb83642caa0a62c1c4d69fba9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680688587699 2023-04-05 09:56:30,359 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 0d0fc1c7b6954f17818cba6203b5debc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688589866 2023-04-05 09:56:30,366 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:30,381 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:30,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:30,384 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:30,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:30,384 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:30,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,385 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,386 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,386 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,387 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/8a66c42e6fbe486d9676224f2d67d4a3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8a66c42e6fbe486d9676224f2d67d4a3 2023-04-05 09:56:30,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:30,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:30,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:30,389 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:30,389 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:30,390 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:30,392 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into 8a66c42e6fbe486d9676224f2d67d4a3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:30,392 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:30,392 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=13, startTime=1680688590356; duration=0sec 2023-04-05 09:56:30,392 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:30,392 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:30,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:30,393 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:30,393 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,394 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/54e57d89de8640739ad99b93250dfcdf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=35.6 K 2023-04-05 09:56:30,394 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 54e57d89de8640739ad99b93250dfcdf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:30,395 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 6fd2fbebedbb4b5f970ab86b4048bcba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680688587699 2023-04-05 09:56:30,395 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting cf1393fe89964aa1ae753be75f87d869, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688589866 2023-04-05 09:56:30,401 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#533 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:30,404 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040564235acdd9824f67b3cc2a6531c0de1f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564235acdd9824f67b3cc2a6531c0de1f 2023-04-05 09:56:30,411 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/2005307272894994880bb3209980cb47 2023-04-05 09:56:30,413 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/f0367fe95f2a4baca8db22fa95328b88 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/f0367fe95f2a4baca8db22fa95328b88 2023-04-05 09:56:30,428 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into f0367fe95f2a4baca8db22fa95328b88(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:30,428 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:30,428 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=13, startTime=1680688590356; duration=0sec 2023-04-05 09:56:30,428 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:30,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:30,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688650523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688650524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688650527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688650555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688650559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,573 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c9da1543184a4905a7808a0672f64354 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c9da1543184a4905a7808a0672f64354 2023-04-05 09:56:30,579 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/A of a168ad47131a50498eee7b0d12023d0b into c9da1543184a4905a7808a0672f64354(size=25.7 K), total size for store is 103.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 09:56:30,579 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:30,579 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/A, priority=9, startTime=1680688589321; duration=1sec 2023-04-05 09:56:30,579 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:30,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39350 deadline: 1680688650629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688650629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688650632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688650833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,837 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/50576901cd37499e831780c1cea366af 2023-04-05 09:56:30,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688650833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688650834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,845 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/ac8333b832af42ce916b2eb5734551bc 2023-04-05 09:56:30,849 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/2005307272894994880bb3209980cb47 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47 2023-04-05 09:56:30,852 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47, entries=150, sequenceid=229, filesize=25.3 K 2023-04-05 09:56:30,853 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/50576901cd37499e831780c1cea366af as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af 2023-04-05 09:56:30,857 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af, entries=150, sequenceid=229, filesize=11.8 K 2023-04-05 09:56:30,858 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/ac8333b832af42ce916b2eb5734551bc as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc 2023-04-05 09:56:30,863 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc, entries=150, sequenceid=229, filesize=11.8 K 2023-04-05 09:56:30,864 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for a168ad47131a50498eee7b0d12023d0b in 474ms, sequenceid=229, compaction requested=true 2023-04-05 09:56:30,864 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:30,864 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:30,865 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55899ffe[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:30,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:30,868 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,869 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:30,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:30,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,869 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:30,869 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:30,871 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:30,874 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:30,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:30,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:30,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:30,875 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:30,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,877 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:30,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,879 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,881 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,888 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,888 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:30,888 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:30,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:30,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,888 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,888 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:30,888 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:30,889 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:30,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:30,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:30,889 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,989 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:30,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:30,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:30,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:30,996 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:30,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:30,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:30,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:30,996 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:30,997 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:30,999 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:30,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:30,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:30,999 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:31,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:31,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:31,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:31,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:31,019 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:31,019 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:31,020 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:31,054 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304059b7795a7a7ce4f25b69de8793e18be45 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b7795a7a7ce4f25b69de8793e18be45 2023-04-05 09:56:31,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:31,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688591087, max timestamp=1680688591096Num uniques 859; 2023-04-05 09:56:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:31,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688591087, max timestamp=1680688591096Num uniques 859; 2023-04-05 09:56:31,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:31,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688591087, max timestamp=1680688591096Num uniques 859; 2023-04-05 09:56:31,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:31,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688651131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688651133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688651138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688651139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688651140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688651235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688651237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688651438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688651442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,467 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/ecabca5d5f6046c5818aafdfa8cb5c42 2023-04-05 09:56:31,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688651642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688651645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688651645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688651741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:31,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688651745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:31,876 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/b85d0054137d4dd7b1a05923b7880ad7 2023-04-05 09:56:32,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688652248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688652249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,289 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/d638204f9c8e48528e56637de96d43ff 2023-04-05 09:56:32,295 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/ecabca5d5f6046c5818aafdfa8cb5c42 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42 2023-04-05 09:56:32,299 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42, entries=150, sequenceid=243, filesize=25.3 K 2023-04-05 09:56:32,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/b85d0054137d4dd7b1a05923b7880ad7 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7 2023-04-05 09:56:32,303 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7, entries=150, sequenceid=243, filesize=11.8 K 2023-04-05 09:56:32,304 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/d638204f9c8e48528e56637de96d43ff as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff 2023-04-05 09:56:32,307 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff, entries=150, sequenceid=243, filesize=11.8 K 2023-04-05 09:56:32,307 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a168ad47131a50498eee7b0d12023d0b in 1288ms, sequenceid=243, compaction requested=true 2023-04-05 09:56:32,308 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:32,308 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:32,309 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5657633b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:32,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:32,311 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,313 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:32,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,314 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:32,315 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:32,317 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:32,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:32,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:32,317 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:32,317 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:32,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:32,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,319 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:32,319 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,320 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,322 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:32,322 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:32,322 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:32,322 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:32,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:32,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:32,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:32,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:32,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:32,323 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,323 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:32,323 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:32,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:32,423 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:32,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:32,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:32,427 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:32,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:32,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:32,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:32,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,427 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:32,429 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,429 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:32,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:32,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,430 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:32,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:32,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:32,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:32,431 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:32,431 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:32,431 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:32,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:32,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688652663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688652664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688652665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688652765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688652765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,843 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f64647b84b514e639b0b4890259e5119 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f64647b84b514e639b0b4890259e5119 2023-04-05 09:56:32,849 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c5c3b42d4ee64ba580351bdc64156cc9 2023-04-05 09:56:32,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688652967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:32,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:32,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688652972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39330 deadline: 1680688653252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,263 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/fa0d671447b0479bb80e518eb4997c8c 2023-04-05 09:56:33,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39378 deadline: 1680688653261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688653271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688653277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,690 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/a2094adc338042fcb6225015a658a66b 2023-04-05 09:56:33,694 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c5c3b42d4ee64ba580351bdc64156cc9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9 2023-04-05 09:56:33,697 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9, entries=150, sequenceid=279, filesize=25.4 K 2023-04-05 09:56:33,698 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/fa0d671447b0479bb80e518eb4997c8c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c 2023-04-05 09:56:33,701 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c, entries=150, sequenceid=279, filesize=11.9 K 2023-04-05 09:56:33,702 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/a2094adc338042fcb6225015a658a66b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b 2023-04-05 09:56:33,705 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b, entries=150, sequenceid=279, filesize=11.9 K 2023-04-05 09:56:33,706 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a168ad47131a50498eee7b0d12023d0b in 1275ms, sequenceid=279, compaction requested=true 2023-04-05 09:56:33,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:33,706 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,707 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf92552[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:33,710 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:33,710 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,711 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:33,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:33,711 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:33,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,712 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:33,713 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:33,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:33,716 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:33,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:33,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:33,716 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:33,716 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:33,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,719 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,719 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,721 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,724 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,725 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:33,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:33,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,725 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:33,725 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,725 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:33,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:33,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:33,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:33,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688653813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688653816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,825 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:33,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:33,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:33,828 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:33,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:33,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:33,828 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:33,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,829 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,830 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:33,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:33,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,831 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:33,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:33,832 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:33,832 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:33,832 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:33,832 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,833 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:33,833 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:33,834 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,834 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,834 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,834 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c17c330[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:33,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:33,835 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,836 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:33,836 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,837 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:33,837 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:33,838 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:33,840 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:33,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:33,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:33,840 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:33,840 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:33,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,842 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,843 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,843 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,844 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:33,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,848 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:33,847 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,848 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:33,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:33,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,848 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688653917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:33,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688653920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,948 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:33,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:33,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:33,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:33,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:33,951 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:33,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:33,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:33,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:33,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,951 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,952 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,952 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:33,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,953 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:33,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:33,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:33,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:33,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:33,954 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,960 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6578d523[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:33,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:33,962 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,963 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:33,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,963 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:33,964 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:33,967 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:33,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:33,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:33,967 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:33,967 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:33,969 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,969 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,969 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:33,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:33,969 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,970 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:33,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:33,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,973 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:33,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:33,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:33,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:33,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:33,973 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:33,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:33,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:33,974 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:33,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,974 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:33,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:33,974 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,074 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:34,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:34,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:34,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:34,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:34,080 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:34,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:34,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:34,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,080 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,082 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,082 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,083 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:34,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:34,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:34,087 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85b2c0c[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:34,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:34,089 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,092 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,092 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:34,094 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:34,099 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:34,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:34,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:34,100 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:34,100 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:34,102 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,102 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,102 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,102 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,104 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,107 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:34,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:34,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,107 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:34,107 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:34,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:34,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688654123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688654123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,192 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405358f549b4f544a48b1887acd21aaf9da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405358f549b4f544a48b1887acd21aaf9da 2023-04-05 09:56:34,207 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:34,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:34,210 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/d9974b88377149e0895696ae3cf08af5 2023-04-05 09:56:34,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:34,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:34,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:34,230 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:34,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:34,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:34,230 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,231 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,231 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,232 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,233 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:34,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:34,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:34,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:34,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:34,256 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:34,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:34,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:34,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/a841c60f81a14e459af7593c260d85c9 2023-04-05 09:56:34,256 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,258 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aca4be0[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:34,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:34,260 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,261 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,262 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:34,264 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,264 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:34,264 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,264 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:34,267 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:34,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:34,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:34,267 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:34,267 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:34,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,268 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,269 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,270 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,274 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:34,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:34,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,274 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,274 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:34,274 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,275 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:34,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/b6a2f5ca561849e6bb14d3753cf74dc5 2023-04-05 09:56:34,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/d9974b88377149e0895696ae3cf08af5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5 2023-04-05 09:56:34,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5, entries=200, sequenceid=291, filesize=32.3 K 2023-04-05 09:56:34,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/a841c60f81a14e459af7593c260d85c9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9 2023-04-05 09:56:34,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9, entries=150, sequenceid=291, filesize=11.9 K 2023-04-05 09:56:34,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/b6a2f5ca561849e6bb14d3753cf74dc5 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5 2023-04-05 09:56:34,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5, entries=150, sequenceid=291, filesize=11.9 K 2023-04-05 09:56:34,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a168ad47131a50498eee7b0d12023d0b in 548ms, sequenceid=291, compaction requested=true 2023-04-05 09:56:34,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:34,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:34,327 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 09:56:34,327 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:34,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:34,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:34,329 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 216952 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 09:56:34,329 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/A is initiating minor compaction (all files) 2023-04-05 09:56:34,329 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/A in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,329 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c9da1543184a4905a7808a0672f64354, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=211.9 K 2023-04-05 09:56:34,329 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60958 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:34,329 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:34,330 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,330 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8a66c42e6fbe486d9676224f2d67d4a3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=59.5 K 2023-04-05 09:56:34,330 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c9da1543184a4905a7808a0672f64354, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680688587511 2023-04-05 09:56:34,330 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8a66c42e6fbe486d9676224f2d67d4a3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688589866 2023-04-05 09:56:34,332 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting fe847d9bb5ba4f20a8194250b2d7b62d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680688587699 2023-04-05 09:56:34,333 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 4de6869013604bdf8e2ba8de21b126dd, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688589834 2023-04-05 09:56:34,333 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2005307272894994880bb3209980cb47, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680688589937 2023-04-05 09:56:34,333 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 50576901cd37499e831780c1cea366af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680688589937 2023-04-05 09:56:34,333 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting ecabca5d5f6046c5818aafdfa8cb5c42, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688590522 2023-04-05 09:56:34,334 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b85d0054137d4dd7b1a05923b7880ad7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688590522 2023-04-05 09:56:34,334 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting c5c3b42d4ee64ba580351bdc64156cc9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680688591127 2023-04-05 09:56:34,334 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting fa0d671447b0479bb80e518eb4997c8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680688591127 2023-04-05 09:56:34,334 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting d9974b88377149e0895696ae3cf08af5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:34,335 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a841c60f81a14e459af7593c260d85c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:34,347 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#545 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:34,375 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:34,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:34,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:34,407 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#A#compaction#546 average throughput is 0.39 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:34,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:34,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:34,408 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:34,408 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,408 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:34,408 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:34,412 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,412 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,413 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,413 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:34,415 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/b1f052e38f324de39504b45e8d69b37e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b1f052e38f324de39504b45e8d69b37e 2023-04-05 09:56:34,419 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into b1f052e38f324de39504b45e8d69b37e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:34,419 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:34,419 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=11, startTime=1680688594327; duration=0sec 2023-04-05 09:56:34,419 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:34,419 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 09:56:34,420 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60958 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 09:56:34,420 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:34,421 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,421 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/f0367fe95f2a4baca8db22fa95328b88, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=59.5 K 2023-04-05 09:56:34,421 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting f0367fe95f2a4baca8db22fa95328b88, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680688589866 2023-04-05 09:56:34,421 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting ac8333b832af42ce916b2eb5734551bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680688589937 2023-04-05 09:56:34,422 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d638204f9c8e48528e56637de96d43ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680688590522 2023-04-05 09:56:34,422 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a2094adc338042fcb6225015a658a66b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680688591127 2023-04-05 09:56:34,422 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b6a2f5ca561849e6bb14d3753cf74dc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:34,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:34,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:34,432 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:34,432 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:34,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,432 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:34,432 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:34,433 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,435 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#547 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:34,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:34,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:34,468 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/a3bc0979828e4c4ea3a6a7377a3ae58e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a3bc0979828e4c4ea3a6a7377a3ae58e 2023-04-05 09:56:34,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688654465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39322 deadline: 1680688654469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,474 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into a3bc0979828e4c4ea3a6a7377a3ae58e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:34,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:34,474 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=11, startTime=1680688594327; duration=0sec 2023-04-05 09:56:34,474 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:34,483 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cb818cfc51ac4661867abc0ad73b5ab3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb818cfc51ac4661867abc0ad73b5ab3 2023-04-05 09:56:34,534 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/6f58a28b041445c8a7d3e37d129e2632 2023-04-05 09:56:34,568 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/d33910a96c0a474ca8a2e387688aeffb 2023-04-05 09:56:34,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688654570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39322 deadline: 1680688654574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,586 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/eeb349c71a0e410bb6aaf86e5f73e766 2023-04-05 09:56:34,590 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/6f58a28b041445c8a7d3e37d129e2632 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632 2023-04-05 09:56:34,594 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632, entries=150, sequenceid=312, filesize=25.4 K 2023-04-05 09:56:34,596 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/d33910a96c0a474ca8a2e387688aeffb as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb 2023-04-05 09:56:34,599 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb, entries=150, sequenceid=312, filesize=11.9 K 2023-04-05 09:56:34,600 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/eeb349c71a0e410bb6aaf86e5f73e766 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766 2023-04-05 09:56:34,604 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766, entries=150, sequenceid=312, filesize=11.9 K 2023-04-05 09:56:34,605 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a168ad47131a50498eee7b0d12023d0b in 172ms, sequenceid=312, compaction requested=false 2023-04-05 09:56:34,605 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:34,605 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:34,606 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,608 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,608 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8060fca[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,621 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:34,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:34,621 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:34,623 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:34,626 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:34,627 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:34,627 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:34,627 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:34,627 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:34,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,628 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,631 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,635 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,635 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:34,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,635 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,635 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:34,635 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,635 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:34,635 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,636 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:34,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:34,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:34,706 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405281c54ccc8c248b48eae85dbac2e0fca to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405281c54ccc8c248b48eae85dbac2e0fca 2023-04-05 09:56:34,710 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/33e78f9addea4e31a400d67d6d5a8d86 2023-04-05 09:56:34,736 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:34,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:34,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:34,739 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:34,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:34,739 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:34,740 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,740 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,741 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:34,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:34,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:34,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:34,743 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,743 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:34,743 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:34,743 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:34,745 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,745 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b740224[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:34,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:34,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,749 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:34,750 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:34,750 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:34,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,750 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:34,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:34,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,752 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:34,757 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:34,759 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:34,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:34,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:34,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:34,759 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:34,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,762 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,763 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,763 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688654760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,766 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:34,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:34,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,766 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,766 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:34,766 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,768 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688654775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39322 deadline: 1680688654779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,866 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:34,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:34,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:34,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688654866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:34,874 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:34,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:34,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:34,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,874 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,875 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,876 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,876 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:34,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:34,880 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:34,880 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:34,880 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:34,881 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/2c5d1184f4cc4cd9b500b57626a8a292 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2c5d1184f4cc4cd9b500b57626a8a292 2023-04-05 09:56:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:34,881 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,881 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:34,882 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:34,882 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:34,883 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,883 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9e700a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:34,886 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:34,886 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,890 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:34,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:34,890 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:34,890 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,890 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:34,891 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/A of a168ad47131a50498eee7b0d12023d0b into 2c5d1184f4cc4cd9b500b57626a8a292(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:34,891 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:34,891 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/A, priority=9, startTime=1680688594327; duration=0sec 2023-04-05 09:56:34,891 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:34,892 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:34,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:34,894 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:34,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:34,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:34,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:34,895 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:34,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:34,896 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,896 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:34,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,898 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,901 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:34,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:34,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:34,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:34,902 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:34,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:34,902 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:34,902 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:34,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,002 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,005 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,005 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,006 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,007 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,007 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,007 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,007 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,009 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,009 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,009 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,009 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,010 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6213b1f2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,012 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,012 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,013 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,013 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,017 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,017 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,020 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,020 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,022 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,026 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,025 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,026 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,026 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688655070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688655078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39322 deadline: 1680688655083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/2eb2d9dcee84445e977c071910b5ae82 2023-04-05 09:56:35,126 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,130 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,130 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,132 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,132 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,133 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,136 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,136 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,136 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,136 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,138 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad38a25[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,140 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,142 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,143 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,144 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,148 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,148 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4e1aba050cff40bda3aa3532b964a7e2 2023-04-05 09:56:35,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,150 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,151 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,153 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/33e78f9addea4e31a400d67d6d5a8d86 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86 2023-04-05 09:56:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,157 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,157 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,158 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,158 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,158 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86, entries=150, sequenceid=326, filesize=25.4 K 2023-04-05 09:56:35,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/2eb2d9dcee84445e977c071910b5ae82 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82 2023-04-05 09:56:35,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82, entries=100, sequenceid=326, filesize=9.5 K 2023-04-05 09:56:35,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4e1aba050cff40bda3aa3532b964a7e2 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2 2023-04-05 09:56:35,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2, entries=100, sequenceid=326, filesize=9.5 K 2023-04-05 09:56:35,167 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a168ad47131a50498eee7b0d12023d0b in 481ms, sequenceid=326, compaction requested=true 2023-04-05 09:56:35,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:35,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:35,168 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:35,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:35,168 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:35,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:35,169 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:35,169 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/A is initiating minor compaction (all files) 2023-04-05 09:56:35,169 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/A in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,169 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2c5d1184f4cc4cd9b500b57626a8a292, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=76.9 K 2023-04-05 09:56:35,169 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:35,169 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:35,169 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,169 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 2c5d1184f4cc4cd9b500b57626a8a292, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:35,170 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b1f052e38f324de39504b45e8d69b37e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=33.9 K 2023-04-05 09:56:35,170 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting b1f052e38f324de39504b45e8d69b37e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:35,170 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 6f58a28b041445c8a7d3e37d129e2632, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680688593811 2023-04-05 09:56:35,171 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting d33910a96c0a474ca8a2e387688aeffb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680688593811 2023-04-05 09:56:35,171 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 33e78f9addea4e31a400d67d6d5a8d86, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688594463 2023-04-05 09:56:35,171 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 2eb2d9dcee84445e977c071910b5ae82, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688594467 2023-04-05 09:56:35,179 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:35,183 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#A#compaction#554 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:35,210 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/20c2042675074224b60c79297dae1d1e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/20c2042675074224b60c79297dae1d1e 2023-04-05 09:56:35,215 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into 20c2042675074224b60c79297dae1d1e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:35,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:35,215 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=13, startTime=1680688595168; duration=0sec 2023-04-05 09:56:35,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:35,215 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 09:56:35,216 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 09:56:35,216 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:35,216 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,217 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a3bc0979828e4c4ea3a6a7377a3ae58e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=33.9 K 2023-04-05 09:56:35,217 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a3bc0979828e4c4ea3a6a7377a3ae58e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680688592652 2023-04-05 09:56:35,217 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting eeb349c71a0e410bb6aaf86e5f73e766, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680688593811 2023-04-05 09:56:35,218 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4e1aba050cff40bda3aa3532b964a7e2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688594467 2023-04-05 09:56:35,226 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#556 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:35,237 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/310cbb6f8b0b459d9059d1d6cfeb85c9 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/310cbb6f8b0b459d9059d1d6cfeb85c9 2023-04-05 09:56:35,241 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into 310cbb6f8b0b459d9059d1d6cfeb85c9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:35,241 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:35,241 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=13, startTime=1680688595168; duration=0sec 2023-04-05 09:56:35,241 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:35,258 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,264 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,264 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,265 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:35,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 09:56:35,266 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:35,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:35,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:35,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,266 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,268 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,268 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,268 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,268 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,270 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,271 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513dc679[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,276 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,278 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,279 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,280 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,280 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,282 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,282 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,283 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,283 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,284 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688655282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,285 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,286 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688655284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,289 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,289 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,290 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,290 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39398 deadline: 1680688655372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688655385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688655387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,390 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,395 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,395 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,395 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,395 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,396 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,396 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,397 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,397 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,397 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,398 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,398 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,399 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,399 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,399 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,399 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,401 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,402 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582ae274[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,403 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,404 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,404 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,404 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,404 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,405 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,406 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,408 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,409 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,409 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,410 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,410 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,410 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,411 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,412 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,418 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,419 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,419 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,418 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,519 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,522 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,522 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,523 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,527 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,527 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,528 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,528 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,529 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,529 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,529 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,529 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,531 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c944a2[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,533 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,534 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,534 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,534 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,535 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,535 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,535 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,535 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,538 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,538 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,538 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,541 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,541 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,541 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,541 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,542 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,544 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,544 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,545 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,545 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,545 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688655581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39322 deadline: 1680688655588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688655590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688655591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,645 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,649 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,649 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,651 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,651 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,654 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,654 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,655 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,655 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618d70dc[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,657 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,658 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,658 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,658 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,658 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,661 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,661 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,664 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,664 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,664 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,665 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,666 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,666 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,668 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,670 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,670 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,671 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,671 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,692 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f61a91f8d8ff404dae7c9a077fd3d377 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f61a91f8d8ff404dae7c9a077fd3d377 2023-04-05 09:56:35,698 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/b273ed33379c490d80307d438fa3282f 2023-04-05 09:56:35,771 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,774 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,774 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,775 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,775 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,776 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,777 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,777 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,777 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,777 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,779 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,779 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebc7c9a[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,780 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,781 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,782 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,782 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,784 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,786 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,786 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,786 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,788 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,788 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,789 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,791 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,791 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,792 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,791 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,792 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,792 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39398 deadline: 1680688655878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,892 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:35,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:35,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688655893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:35,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:35,895 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:35,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:35,895 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:35,896 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,896 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:35,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688655895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,900 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,900 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:35,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:35,903 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510a0c7b[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:35,905 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:35,905 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,906 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,906 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:35,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:35,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:35,906 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:35,908 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:35,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,908 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:35,910 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:35,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:35,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:35,910 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:35,910 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:35,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:35,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,913 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:35,914 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,917 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,917 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:35,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:35,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:35,917 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:35,918 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:35,918 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:35,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:35,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:35,917 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:35,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:35,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:35,918 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:35,917 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:35,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:35,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:35,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,018 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,021 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,022 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,023 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,023 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,024 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,025 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,025 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,025 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:36,025 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,026 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,026 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,027 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,027 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c0baeac[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,029 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,032 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,032 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,032 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,033 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,036 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,036 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,036 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,037 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,038 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,039 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,041 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:36,041 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,042 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,042 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,042 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,042 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/4f490417152946d1bc17685c13b6e662 2023-04-05 09:56:36,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/8f7cedab04734ca4a4f937e4cd0050ae 2023-04-05 09:56:36,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/b273ed33379c490d80307d438fa3282f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f 2023-04-05 09:56:36,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f, entries=200, sequenceid=349, filesize=32.3 K 2023-04-05 09:56:36,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/4f490417152946d1bc17685c13b6e662 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662 2023-04-05 09:56:36,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662, entries=150, sequenceid=349, filesize=11.9 K 2023-04-05 09:56:36,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/8f7cedab04734ca4a4f937e4cd0050ae as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae 2023-04-05 09:56:36,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae, entries=150, sequenceid=349, filesize=11.9 K 2023-04-05 09:56:36,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a168ad47131a50498eee7b0d12023d0b in 871ms, sequenceid=349, compaction requested=false 2023-04-05 09:56:36,136 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,142 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,145 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,145 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,147 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,147 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,148 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,149 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,149 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:36,149 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,160 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f25a01eb7a5c4be688e397d6417da289 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f25a01eb7a5c4be688e397d6417da289 2023-04-05 09:56:36,164 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=361, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/62849cb01dc84a0aaeeaefd2e70f541e 2023-04-05 09:56:36,170 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3086233d989048d9946f7c2df3456300 2023-04-05 09:56:36,191 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/a2a5e40c59524faa84ef9b6b1f1195ab 2023-04-05 09:56:36,194 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/62849cb01dc84a0aaeeaefd2e70f541e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e 2023-04-05 09:56:36,197 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e, entries=150, sequenceid=361, filesize=25.4 K 2023-04-05 09:56:36,198 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/3086233d989048d9946f7c2df3456300 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300 2023-04-05 09:56:36,201 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300, entries=150, sequenceid=361, filesize=11.9 K 2023-04-05 09:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/a2a5e40c59524faa84ef9b6b1f1195ab as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab 2023-04-05 09:56:36,204 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab, entries=150, sequenceid=361, filesize=11.9 K 2023-04-05 09:56:36,205 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for a168ad47131a50498eee7b0d12023d0b in 56ms, sequenceid=361, compaction requested=true 2023-04-05 09:56:36,205 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,205 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,205 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,207 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,207 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360a3261[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,209 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,209 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,210 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,210 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,211 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,214 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,214 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,214 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,217 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,218 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,219 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,221 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,221 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,222 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,221 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,322 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,342 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,342 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,346 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,346 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,346 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,346 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,348 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,348 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,349 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,349 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,349 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,352 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,352 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,354 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@befc096[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,357 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,358 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,358 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,358 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,358 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,360 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,360 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,363 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,363 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,363 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,364 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,365 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,366 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,370 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:36,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,370 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,370 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,370 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:36,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:36,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:36,437 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/beddf3769c1a4fa484a1b07313a09459 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/beddf3769c1a4fa484a1b07313a09459 2023-04-05 09:56:36,441 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/A of a168ad47131a50498eee7b0d12023d0b into beddf3769c1a4fa484a1b07313a09459(size=26.1 K), total size for store is 83.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 09:56:36,442 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,442 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/A, priority=13, startTime=1680688595168; duration=1sec 2023-04-05 09:56:36,442 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:36,444 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040538b7536c4dab4d089adfa2c54c7efdba to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040538b7536c4dab4d089adfa2c54c7efdba 2023-04-05 09:56:36,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688656462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688596451, max timestamp=1680688596462Num uniques 859; 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688596451, max timestamp=1680688596462Num uniques 859; 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680688596451, max timestamp=1680688596462Num uniques 859; 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 09:56:36,471 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,473 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,473 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,474 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,475 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,475 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,476 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,478 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,478 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,478 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:36,478 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,480 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a076c44[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,483 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,484 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,484 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,485 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,487 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,490 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,490 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,490 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,490 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,490 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,492 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,492 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,492 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,493 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,495 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,497 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,497 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,497 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,497 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688656521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688656565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39350 deadline: 1680688656588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,597 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39322 deadline: 1680688656596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,600 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,601 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,602 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,602 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,607 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,607 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,607 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,607 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:36,607 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,610 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b18dd66[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,614 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,615 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,615 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,616 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,620 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,621 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,622 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,624 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,625 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688656626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,629 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,629 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,629 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,629 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,729 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,736 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,737 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,737 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,738 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,738 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,739 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,739 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,739 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:36,739 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,741 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce6735e[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,742 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,743 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,747 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,748 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,749 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,749 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,752 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,752 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,752 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,754 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,754 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,759 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,761 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,761 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,761 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,761 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688656768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688656830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,848 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/1a86c94f5844443aa8c5c134d38f0d7f 2023-04-05 09:56:36,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/063fc671491e4c3ba6682cc411d16df0 2023-04-05 09:56:36,861 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:36,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:36,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:36,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:36,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39398 deadline: 1680688656882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:36,888 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:36,888 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,888 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:36,888 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:36,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,889 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/523a3fa8502b4dd2b3f20668d42cb69e 2023-04-05 09:56:36,891 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,891 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,891 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,892 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:36,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:36,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:36,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:36,894 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:36,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:36,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:36,895 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:36,895 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:36,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/1a86c94f5844443aa8c5c134d38f0d7f as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f 2023-04-05 09:56:36,897 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f, entries=400, sequenceid=373, filesize=59.8 K 2023-04-05 09:56:36,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:36,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f8e924[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:36,901 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:36,901 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/063fc671491e4c3ba6682cc411d16df0 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0 2023-04-05 09:56:36,902 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:36,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:36,903 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:36,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,903 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:36,904 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0, entries=150, sequenceid=373, filesize=11.9 K 2023-04-05 09:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/523a3fa8502b4dd2b3f20668d42cb69e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e 2023-04-05 09:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:36,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:36,908 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:36,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:36,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:36,909 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:36,909 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:36,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e, entries=150, sequenceid=373, filesize=11.9 K 2023-04-05 09:56:36,911 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:36,911 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a168ad47131a50498eee7b0d12023d0b in 479ms, sequenceid=373, compaction requested=true 2023-04-05 09:56:36,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,911 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:36,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,912 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,912 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:36,912 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:36,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:36,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,913 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,913 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,913 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/B is initiating minor compaction (all files) 2023-04-05 09:56:36,913 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 147148 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:36,913 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/B in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,913 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/A is initiating minor compaction (all files) 2023-04-05 09:56:36,913 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/20c2042675074224b60c79297dae1d1e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=48.3 K 2023-04-05 09:56:36,913 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/A in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:36,914 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/beddf3769c1a4fa484a1b07313a09459, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=143.7 K 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 20c2042675074224b60c79297dae1d1e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688593811 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting beddf3769c1a4fa484a1b07313a09459, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688593811 2023-04-05 09:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 4f490417152946d1bc17685c13b6e662, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680688594754 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting b273ed33379c490d80307d438fa3282f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680688594728 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 3086233d989048d9946f7c2df3456300, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688595280 2023-04-05 09:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,914 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 62849cb01dc84a0aaeeaefd2e70f541e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688595280 2023-04-05 09:56:36,915 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 063fc671491e4c3ba6682cc411d16df0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680688596426 2023-04-05 09:56:36,915 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] compactions.Compactor(207): Compacting 1a86c94f5844443aa8c5c134d38f0d7f, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680688596415 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,915 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:36,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:36,915 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:36,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:36,916 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:36,915 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:36,916 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:36,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:36,921 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#B#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:36,923 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#A#compaction#567 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:36,935 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/369c730a42854bcaa22a9aa769537364 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/369c730a42854bcaa22a9aa769537364 2023-04-05 09:56:36,939 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/B of a168ad47131a50498eee7b0d12023d0b into 369c730a42854bcaa22a9aa769537364(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:36,940 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:36,940 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/B, priority=12, startTime=1680688596912; duration=0sec 2023-04-05 09:56:36,940 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 09:56:36,940 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 09:56:36,941 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 09:56:36,941 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1912): a168ad47131a50498eee7b0d12023d0b/C is initiating minor compaction (all files) 2023-04-05 09:56:36,941 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a168ad47131a50498eee7b0d12023d0b/C in TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:36,941 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/310cbb6f8b0b459d9059d1d6cfeb85c9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e] into tmpdir=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp, totalSize=48.3 K 2023-04-05 09:56:36,942 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 310cbb6f8b0b459d9059d1d6cfeb85c9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680688593811 2023-04-05 09:56:36,942 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 8f7cedab04734ca4a4f937e4cd0050ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680688594754 2023-04-05 09:56:36,942 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting a2a5e40c59524faa84ef9b6b1f1195ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680688595280 2023-04-05 09:56:36,943 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] compactions.Compactor(207): Compacting 523a3fa8502b4dd2b3f20668d42cb69e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680688596426 2023-04-05 09:56:36,949 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] throttle.PressureAwareThroughputController(145): a168ad47131a50498eee7b0d12023d0b#C#compaction#568 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 09:56:37,016 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:37,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:37,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:37,019 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:37,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:37,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:37,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:37,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:37,019 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:37,020 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:37,020 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:37,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,021 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:37,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:37,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:37,022 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:37,022 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:37,022 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:37,022 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:37,022 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:37,022 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:37,022 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:37,022 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 09:56:37,022 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:37,022 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:37,023 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:37,023 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:37,023 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:37,023 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 09:56:37,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:37,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:37,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688657095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688657135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688657198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,346 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/c3f5ea0787ee40b9923e578953b37a74 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c3f5ea0787ee40b9923e578953b37a74 2023-04-05 09:56:37,350 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/A of a168ad47131a50498eee7b0d12023d0b into c3f5ea0787ee40b9923e578953b37a74(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:37,350 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:37,350 INFO [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/A, priority=12, startTime=1680688596912; duration=0sec 2023-04-05 09:56:37,351 DEBUG [RS:0;jenkins-hbase4:46827-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:37,360 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/b06c977d9410476ca672793089dfa750 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b06c977d9410476ca672793089dfa750 2023-04-05 09:56:37,364 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a168ad47131a50498eee7b0d12023d0b/C of a168ad47131a50498eee7b0d12023d0b into b06c977d9410476ca672793089dfa750(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 09:56:37,364 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:37,365 INFO [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b., storeName=a168ad47131a50498eee7b0d12023d0b/C, priority=12, startTime=1680688596912; duration=0sec 2023-04-05 09:56:37,365 DEBUG [RS:0;jenkins-hbase4:46827-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 09:56:37,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688657401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,433 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d82ee2fe9db74c3299c0304fdd1cbb18 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d82ee2fe9db74c3299c0304fdd1cbb18 2023-04-05 09:56:37,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39330 deadline: 1680688657638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:37,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688657705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:37,858 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=411, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7eee40474e2340ea825407358244e8a1 2023-04-05 09:56:37,867 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/01dc9357755e4f31b26e93c9ebd3608d 2023-04-05 09:56:38,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 09:56:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46827] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39378 deadline: 1680688658208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,275 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/2e21a9ae8c8c4708a00950e61fd57dc3 2023-04-05 09:56:38,279 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7eee40474e2340ea825407358244e8a1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7eee40474e2340ea825407358244e8a1 2023-04-05 09:56:38,283 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7eee40474e2340ea825407358244e8a1, entries=150, sequenceid=411, filesize=25.4 K 2023-04-05 09:56:38,284 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/01dc9357755e4f31b26e93c9ebd3608d as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/01dc9357755e4f31b26e93c9ebd3608d 2023-04-05 09:56:38,291 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/01dc9357755e4f31b26e93c9ebd3608d, entries=150, sequenceid=411, filesize=11.9 K 2023-04-05 09:56:38,292 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/2e21a9ae8c8c4708a00950e61fd57dc3 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/2e21a9ae8c8c4708a00950e61fd57dc3 2023-04-05 09:56:38,298 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/2e21a9ae8c8c4708a00950e61fd57dc3, entries=150, sequenceid=411, filesize=11.9 K 2023-04-05 09:56:38,299 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a168ad47131a50498eee7b0d12023d0b in 1277ms, sequenceid=411, compaction requested=false 2023-04-05 09:56:38,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:38,299 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:38,300 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:38,302 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,303 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,303 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48acdf13[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:38,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:38,309 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,312 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:38,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,313 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:38,320 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:38,323 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:38,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:38,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:38,323 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:38,323 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:38,325 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:38,325 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,327 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,330 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:38,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:38,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,330 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:38,330 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:38,331 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:38,331 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,332 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,332 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:38,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,332 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:38,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:38,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:38,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:38,431 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:38,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:38,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 09:56:38,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 09:56:38,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 09:56:38,434 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 09:56:38,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:38,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 09:56:38,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 09:56:38,434 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,435 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:38,436 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:38,436 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:38,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:38,437 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 09:56:38,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 09:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:38,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 09:56:38,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 09:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 09:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 09:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 09:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 09:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 09:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 09:56:38,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:38,440 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. started... 2023-04-05 09:56:38,440 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:38,441 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:38,578 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-05 09:56:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46827] regionserver.HRegion(9158): Flush requested on a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:38,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. as already flushing 2023-04-05 09:56:38,599 DEBUG [Thread-3678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:62753 2023-04-05 09:56:38,599 DEBUG [Thread-3678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,600 DEBUG [Thread-3680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:62753 2023-04-05 09:56:38,600 DEBUG [Thread-3680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,601 DEBUG [Thread-3686] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:62753 2023-04-05 09:56:38,601 DEBUG [Thread-3686] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,601 DEBUG [Thread-3669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:62753 2023-04-05 09:56:38,601 DEBUG [Thread-3669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,602 DEBUG [Thread-3682] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:62753 2023-04-05 09:56:38,602 DEBUG [Thread-3682] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,603 DEBUG [Thread-3684] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:62753 2023-04-05 09:56:38,603 DEBUG [Thread-3684] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,606 DEBUG [Thread-3671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:62753 2023-04-05 09:56:38,606 DEBUG [Thread-3671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,648 DEBUG [Thread-3673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:62753 2023-04-05 09:56:38,648 DEBUG [Thread-3673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:38,864 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040553e51bd6f08048038039ce6c02051417 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553e51bd6f08048038039ce6c02051417 2023-04-05 09:56:38,904 DEBUG [Thread-3675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:62753 2023-04-05 09:56:38,905 DEBUG [Thread-3675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:39,220 DEBUG [Thread-3667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:62753 2023-04-05 09:56:39,220 DEBUG [Thread-3667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:39,268 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=424, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/56303232bbe34bf8b719583347408c84 2023-04-05 09:56:39,676 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/53bac2afbeaa40c297f0a420f2e1a8e1 2023-04-05 09:56:40,086 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/890d01202fcc49e3891a8bb30c1d177c 2023-04-05 09:56:40,089 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/56303232bbe34bf8b719583347408c84 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/56303232bbe34bf8b719583347408c84 2023-04-05 09:56:40,092 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/56303232bbe34bf8b719583347408c84, entries=100, sequenceid=424, filesize=18.6 K 2023-04-05 09:56:40,092 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/53bac2afbeaa40c297f0a420f2e1a8e1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/53bac2afbeaa40c297f0a420f2e1a8e1 2023-04-05 09:56:40,095 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/53bac2afbeaa40c297f0a420f2e1a8e1, entries=100, sequenceid=424, filesize=9.5 K 2023-04-05 09:56:40,095 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/890d01202fcc49e3891a8bb30c1d177c as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/890d01202fcc49e3891a8bb30c1d177c 2023-04-05 09:56:40,097 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/890d01202fcc49e3891a8bb30c1d177c, entries=100, sequenceid=424, filesize=9.5 K 2023-04-05 09:56:40,098 INFO [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=46.96 KB/48090 for a168ad47131a50498eee7b0d12023d0b in 1658ms, sequenceid=424, compaction requested=true 2023-04-05 09:56:40,098 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:40,098 DEBUG [rs(jenkins-hbase4.apache.org,46827,1680688416991)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:40,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 09:56:40,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 09:56:40,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 09:56:40,098 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 09:56:40,103 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 09:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46827,1680688416991' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 09:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f2d88d[Count = 0] remaining members to acquire global barrier 2023-04-05 09:56:40,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 09:56:40,105 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,106 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 09:56:40,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 09:56:40,106 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46827,1680688416991' in zk 2023-04-05 09:56:40,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,106 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 09:56:40,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 09:56:40,107 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:40,107 DEBUG [member: 'jenkins-hbase4.apache.org,46827,1680688416991' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 09:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46827,1680688416991': 2023-04-05 09:56:40,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46827,1680688416991' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 09:56:40,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 09:56:40,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 09:56:40,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 09:56:40,109 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 09:56:40,109 INFO [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 09:56:40,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,111 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 09:56:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 09:56:40,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:40,112 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 09:56:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,113 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 09:56:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 09:56:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 09:56:40,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:40,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 09:56:40,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 09:56:40,114 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 09:56:40,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 09:56:40,115 DEBUG [(jenkins-hbase4.apache.org,34901,1680688415681)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 09:56:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,115 DEBUG [Thread-3677] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 09:56:40,115 DEBUG [Thread-3677] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 09:56:40,115 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 09:56:40,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 09:56:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 09:56:40,215 DEBUG [Thread-3677] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 09:56:40,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4385 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4302 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4291 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4385 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(398): read 4327 2023-04-05 09:56:40,216 INFO [Listener at localhost/42121] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 09:56:40,217 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:56:40,217 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68067de1 to 127.0.0.1:62753 2023-04-05 09:56:40,217 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:40,217 INFO [Listener at localhost/42121] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 09:56:40,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 09:56:40,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:40,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:40,221 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688600220"}]},"ts":"1680688600220"} 2023-04-05 09:56:40,221 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 09:56:40,223 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 09:56:40,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, UNASSIGN}] 2023-04-05 09:56:40,225 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, UNASSIGN 2023-04-05 09:56:40,225 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:40,225 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688600225"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680688600225"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680688600225"}]},"ts":"1680688600225"} 2023-04-05 09:56:40,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991}] 2023-04-05 09:56:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:40,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:40,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a168ad47131a50498eee7b0d12023d0b, disabling compactions & flushes 2023-04-05 09:56:40,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. after waiting 0 ms 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:40,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a168ad47131a50498eee7b0d12023d0b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=A 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=B 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a168ad47131a50498eee7b0d12023d0b, store=C 2023-04-05 09:56:40,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 09:56:40,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:40,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040553dabe34f8af459fb5655a6b4c00b762 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553dabe34f8af459fb5655a6b4c00b762 2023-04-05 09:56:40,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:41,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7485809b277249a5a5bb0b747a7adf98 2023-04-05 09:56:41,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:41,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/d95e7534ef484089893adb60a845851b 2023-04-05 09:56:42,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4025fca302114e908be46c693647b5d8 2023-04-05 09:56:42,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/A/7485809b277249a5a5bb0b747a7adf98 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7485809b277249a5a5bb0b747a7adf98 2023-04-05 09:56:42,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7485809b277249a5a5bb0b747a7adf98, entries=150, sequenceid=434, filesize=25.4 K 2023-04-05 09:56:42,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/B/d95e7534ef484089893adb60a845851b as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d95e7534ef484089893adb60a845851b 2023-04-05 09:56:42,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d95e7534ef484089893adb60a845851b, entries=150, sequenceid=434, filesize=11.9 K 2023-04-05 09:56:42,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/.tmp/C/4025fca302114e908be46c693647b5d8 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4025fca302114e908be46c693647b5d8 2023-04-05 09:56:42,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4025fca302114e908be46c693647b5d8, entries=150, sequenceid=434, filesize=11.9 K 2023-04-05 09:56:42,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for a168ad47131a50498eee7b0d12023d0b in 1642ms, sequenceid=434, compaction requested=true 2023-04-05 09:56:42,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/519cf4af787144978145d6c1482e1660, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c9da1543184a4905a7808a0672f64354, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2c5d1184f4cc4cd9b500b57626a8a292, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/beddf3769c1a4fa484a1b07313a09459, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f] to archive 2023-04-05 09:56:42,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:42,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7163df12b5c8427abd1a3c65f2c8b074 2023-04-05 09:56:42,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e6bed63474ef438c995230d1a46eae95 2023-04-05 09:56:42,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/9878d078fd1e4efc9a93f39a3bb465b7 2023-04-05 09:56:42,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/519cf4af787144978145d6c1482e1660 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/519cf4af787144978145d6c1482e1660 2023-04-05 09:56:42,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ddfe4cdb6c424a3382da2ea7d39ae1b2 2023-04-05 09:56:42,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2b0e87f991774118ac9a640fa39821f9 2023-04-05 09:56:42,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c85e90a2ca4044ca8f0b7e2d5b3faf9b 2023-04-05 09:56:42,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/29e224429dec47b2aa301736992a00ae 2023-04-05 09:56:42,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/22af08849628412e99816a0d8dce6504 2023-04-05 09:56:42,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/e69ff35341484ceb95b74e5c62afcf36 2023-04-05 09:56:42,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f7a18fcffb447b994fd5da89caa7b35 2023-04-05 09:56:42,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c9da1543184a4905a7808a0672f64354 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c9da1543184a4905a7808a0672f64354 2023-04-05 09:56:42,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/fe847d9bb5ba4f20a8194250b2d7b62d 2023-04-05 09:56:42,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/4de6869013604bdf8e2ba8de21b126dd 2023-04-05 09:56:42,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2005307272894994880bb3209980cb47 2023-04-05 09:56:42,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/ecabca5d5f6046c5818aafdfa8cb5c42 2023-04-05 09:56:42,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c5c3b42d4ee64ba580351bdc64156cc9 2023-04-05 09:56:42,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/d9974b88377149e0895696ae3cf08af5 2023-04-05 09:56:42,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2c5d1184f4cc4cd9b500b57626a8a292 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/2c5d1184f4cc4cd9b500b57626a8a292 2023-04-05 09:56:42,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/6f58a28b041445c8a7d3e37d129e2632 2023-04-05 09:56:42,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/beddf3769c1a4fa484a1b07313a09459 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/beddf3769c1a4fa484a1b07313a09459 2023-04-05 09:56:42,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/33e78f9addea4e31a400d67d6d5a8d86 2023-04-05 09:56:42,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/b273ed33379c490d80307d438fa3282f 2023-04-05 09:56:42,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/62849cb01dc84a0aaeeaefd2e70f541e 2023-04-05 09:56:42,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/1a86c94f5844443aa8c5c134d38f0d7f 2023-04-05 09:56:42,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/9656d12cebc14cdf94633bb9f48511cd, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f5d15092c39e42319d6838595ecff981, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8a66c42e6fbe486d9676224f2d67d4a3, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b1f052e38f324de39504b45e8d69b37e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/20c2042675074224b60c79297dae1d1e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0] to archive 2023-04-05 09:56:42,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:42,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/17d18f418f474ae288bda9e7abc3f15f 2023-04-05 09:56:42,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/818eeb38a9364ffaa6fd018fe937daa4 2023-04-05 09:56:42,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a7e60f624576458d81d8974410db5ff5 2023-04-05 09:56:42,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/9656d12cebc14cdf94633bb9f48511cd to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/9656d12cebc14cdf94633bb9f48511cd 2023-04-05 09:56:42,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c79f6ddb39244ffab0784a370d16039d 2023-04-05 09:56:42,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/c15970f2b58e47f5ab3d5a25eba243b0 2023-04-05 09:56:42,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3da2a906a45a42a295d0682f084d4853 2023-04-05 09:56:42,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4004d924ddb34caca322d3912948e5a5 2023-04-05 09:56:42,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f794231a20ef47eb88c4fe49987db981 2023-04-05 09:56:42,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/99bb83634f8b4c0f885e386c9d735212 2023-04-05 09:56:42,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f5d15092c39e42319d6838595ecff981 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/f5d15092c39e42319d6838595ecff981 2023-04-05 09:56:42,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8bda470701b040f19e67423508a432e9 2023-04-05 09:56:42,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3ad48ecdb83642caa0a62c1c4d69fba9 2023-04-05 09:56:42,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8a66c42e6fbe486d9676224f2d67d4a3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/8a66c42e6fbe486d9676224f2d67d4a3 2023-04-05 09:56:42,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/0d0fc1c7b6954f17818cba6203b5debc 2023-04-05 09:56:42,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/50576901cd37499e831780c1cea366af 2023-04-05 09:56:42,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b85d0054137d4dd7b1a05923b7880ad7 2023-04-05 09:56:42,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/fa0d671447b0479bb80e518eb4997c8c 2023-04-05 09:56:42,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b1f052e38f324de39504b45e8d69b37e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/b1f052e38f324de39504b45e8d69b37e 2023-04-05 09:56:42,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/a841c60f81a14e459af7593c260d85c9 2023-04-05 09:56:42,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d33910a96c0a474ca8a2e387688aeffb 2023-04-05 09:56:42,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/20c2042675074224b60c79297dae1d1e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/20c2042675074224b60c79297dae1d1e 2023-04-05 09:56:42,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/2eb2d9dcee84445e977c071910b5ae82 2023-04-05 09:56:42,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/4f490417152946d1bc17685c13b6e662 2023-04-05 09:56:42,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/3086233d989048d9946f7c2df3456300 2023-04-05 09:56:42,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/063fc671491e4c3ba6682cc411d16df0 2023-04-05 09:56:42,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8ac2b3fffbc64b86bb0bf64621a9f0bf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/54e57d89de8640739ad99b93250dfcdf, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/f0367fe95f2a4baca8db22fa95328b88, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a3bc0979828e4c4ea3a6a7377a3ae58e, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/310cbb6f8b0b459d9059d1d6cfeb85c9, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e] to archive 2023-04-05 09:56:42,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 09:56:42,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/01c12a8b34e34aafaad62a5380ae8f6b 2023-04-05 09:56:42,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d38fdc088b9e470aba79963fd6b2b9c1 2023-04-05 09:56:42,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5994c1bc97944c5a83ca20d7c1df4e79 2023-04-05 09:56:42,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8ac2b3fffbc64b86bb0bf64621a9f0bf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8ac2b3fffbc64b86bb0bf64621a9f0bf 2023-04-05 09:56:42,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/15bc53d994b74771865d16d56bdcc550 2023-04-05 09:56:42,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4d805c957fd04e47a104c0fb67a65b99 2023-04-05 09:56:42,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/47dcdc6b3f9b481584bcde5c883677f4 2023-04-05 09:56:42,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fe1d88457f94098b05ed966f1084fb5 2023-04-05 09:56:42,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/31764b1c1f25481da803f5784ae4a065 2023-04-05 09:56:42,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/5c58bbbe663b45b08ce620bfb4d0615f 2023-04-05 09:56:42,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/54e57d89de8640739ad99b93250dfcdf to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/54e57d89de8640739ad99b93250dfcdf 2023-04-05 09:56:42,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/218fdca633d14dc086bb1e789d319ca1 2023-04-05 09:56:42,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/6fd2fbebedbb4b5f970ab86b4048bcba 2023-04-05 09:56:42,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/f0367fe95f2a4baca8db22fa95328b88 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/f0367fe95f2a4baca8db22fa95328b88 2023-04-05 09:56:42,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/cf1393fe89964aa1ae753be75f87d869 2023-04-05 09:56:42,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/ac8333b832af42ce916b2eb5734551bc 2023-04-05 09:56:42,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/d638204f9c8e48528e56637de96d43ff 2023-04-05 09:56:42,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2094adc338042fcb6225015a658a66b 2023-04-05 09:56:42,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a3bc0979828e4c4ea3a6a7377a3ae58e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a3bc0979828e4c4ea3a6a7377a3ae58e 2023-04-05 09:56:42,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b6a2f5ca561849e6bb14d3753cf74dc5 2023-04-05 09:56:42,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/eeb349c71a0e410bb6aaf86e5f73e766 2023-04-05 09:56:42,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/310cbb6f8b0b459d9059d1d6cfeb85c9 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/310cbb6f8b0b459d9059d1d6cfeb85c9 2023-04-05 09:56:42,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4e1aba050cff40bda3aa3532b964a7e2 2023-04-05 09:56:42,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/8f7cedab04734ca4a4f937e4cd0050ae 2023-04-05 09:56:42,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/a2a5e40c59524faa84ef9b6b1f1195ab 2023-04-05 09:56:42,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/523a3fa8502b4dd2b3f20668d42cb69e 2023-04-05 09:56:42,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits/437.seqid, newMaxSeqId=437, maxSeqId=4 2023-04-05 09:56:42,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b. 2023-04-05 09:56:42,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a168ad47131a50498eee7b0d12023d0b: 2023-04-05 09:56:42,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:42,115 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a168ad47131a50498eee7b0d12023d0b, regionState=CLOSED 2023-04-05 09:56:42,116 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680688602115"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680688602115"}]},"ts":"1680688602115"} 2023-04-05 09:56:42,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-05 09:56:42,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure a168ad47131a50498eee7b0d12023d0b, server=jenkins-hbase4.apache.org,46827,1680688416991 in 1.8900 sec 2023-04-05 09:56:42,119 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-05 09:56:42,119 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a168ad47131a50498eee7b0d12023d0b, UNASSIGN in 1.8950 sec 2023-04-05 09:56:42,119 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680688602119"}]},"ts":"1680688602119"} 2023-04-05 09:56:42,120 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 09:56:42,122 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 09:56:42,123 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9040 sec 2023-04-05 09:56:42,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 09:56:42,332 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-05 09:56:42,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 09:56:42,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,335 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,336 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 09:56:42,340 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:42,342 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C, FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits] 2023-04-05 09:56:42,348 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/56303232bbe34bf8b719583347408c84 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/56303232bbe34bf8b719583347408c84 2023-04-05 09:56:42,349 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7485809b277249a5a5bb0b747a7adf98 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7485809b277249a5a5bb0b747a7adf98 2023-04-05 09:56:42,353 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7eee40474e2340ea825407358244e8a1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/7eee40474e2340ea825407358244e8a1 2023-04-05 09:56:42,355 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c3f5ea0787ee40b9923e578953b37a74 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/A/c3f5ea0787ee40b9923e578953b37a74 2023-04-05 09:56:42,358 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/01dc9357755e4f31b26e93c9ebd3608d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/01dc9357755e4f31b26e93c9ebd3608d 2023-04-05 09:56:42,359 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/369c730a42854bcaa22a9aa769537364 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/369c730a42854bcaa22a9aa769537364 2023-04-05 09:56:42,361 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/53bac2afbeaa40c297f0a420f2e1a8e1 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/53bac2afbeaa40c297f0a420f2e1a8e1 2023-04-05 09:56:42,362 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d95e7534ef484089893adb60a845851b to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/B/d95e7534ef484089893adb60a845851b 2023-04-05 09:56:42,364 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/2e21a9ae8c8c4708a00950e61fd57dc3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/2e21a9ae8c8c4708a00950e61fd57dc3 2023-04-05 09:56:42,365 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4025fca302114e908be46c693647b5d8 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/4025fca302114e908be46c693647b5d8 2023-04-05 09:56:42,366 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/890d01202fcc49e3891a8bb30c1d177c to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/890d01202fcc49e3891a8bb30c1d177c 2023-04-05 09:56:42,367 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b06c977d9410476ca672793089dfa750 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/C/b06c977d9410476ca672793089dfa750 2023-04-05 09:56:42,371 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits/437.seqid to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b/recovered.edits/437.seqid 2023-04-05 09:56:42,371 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/.tmp/data/default/TestAcidGuarantees/a168ad47131a50498eee7b0d12023d0b 2023-04-05 09:56:42,372 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 09:56:42,372 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:56:42,374 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 09:56:42,379 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405008ce3b678ab47f2be2a8c6c8c8cc1be to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405008ce3b678ab47f2be2a8c6c8c8cc1be 2023-04-05 09:56:42,380 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050e56fe5bc16c4202a2512ae9f3adcc68 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050e56fe5bc16c4202a2512ae9f3adcc68 2023-04-05 09:56:42,382 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040517d3572bf9ca4572974e6b6ca784e0b7 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040517d3572bf9ca4572974e6b6ca784e0b7 2023-04-05 09:56:42,383 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405281c54ccc8c248b48eae85dbac2e0fca to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405281c54ccc8c248b48eae85dbac2e0fca 2023-04-05 09:56:42,384 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405358f549b4f544a48b1887acd21aaf9da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405358f549b4f544a48b1887acd21aaf9da 2023-04-05 09:56:42,388 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040538b7536c4dab4d089adfa2c54c7efdba to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040538b7536c4dab4d089adfa2c54c7efdba 2023-04-05 09:56:42,389 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054a93a9c83c044a1eb7439fb8ee9c3452 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054a93a9c83c044a1eb7439fb8ee9c3452 2023-04-05 09:56:42,391 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054f8770eb7c3546aa9b92649380958cec to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054f8770eb7c3546aa9b92649380958cec 2023-04-05 09:56:42,392 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553dabe34f8af459fb5655a6b4c00b762 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553dabe34f8af459fb5655a6b4c00b762 2023-04-05 09:56:42,393 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553e51bd6f08048038039ce6c02051417 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040553e51bd6f08048038039ce6c02051417 2023-04-05 09:56:42,394 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564235acdd9824f67b3cc2a6531c0de1f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040564235acdd9824f67b3cc2a6531c0de1f 2023-04-05 09:56:42,396 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580c3e6748d174e6d9f57210ccbf63dc0 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580c3e6748d174e6d9f57210ccbf63dc0 2023-04-05 09:56:42,397 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058ae644903a204213b49af580f9b0733f to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058ae644903a204213b49af580f9b0733f 2023-04-05 09:56:42,398 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f3cec1b8d6f4954a82194e36a2ea439 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f3cec1b8d6f4954a82194e36a2ea439 2023-04-05 09:56:42,399 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040598930516432d4272804dd3687e9432da to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040598930516432d4272804dd3687e9432da 2023-04-05 09:56:42,401 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059ae4055b655e4e0388784a5a90603f1d to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059ae4055b655e4e0388784a5a90603f1d 2023-04-05 09:56:42,403 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b7795a7a7ce4f25b69de8793e18be45 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b7795a7a7ce4f25b69de8793e18be45 2023-04-05 09:56:42,405 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bb65a410b9144d21b71795b5c1abc248 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bb65a410b9144d21b71795b5c1abc248 2023-04-05 09:56:42,406 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb818cfc51ac4661867abc0ad73b5ab3 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cb818cfc51ac4661867abc0ad73b5ab3 2023-04-05 09:56:42,408 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d82ee2fe9db74c3299c0304fdd1cbb18 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d82ee2fe9db74c3299c0304fdd1cbb18 2023-04-05 09:56:42,409 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dcf0b0bd6a5b4fc7b934164fb5637708 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dcf0b0bd6a5b4fc7b934164fb5637708 2023-04-05 09:56:42,411 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f25a01eb7a5c4be688e397d6417da289 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f25a01eb7a5c4be688e397d6417da289 2023-04-05 09:56:42,412 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f61a91f8d8ff404dae7c9a077fd3d377 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f61a91f8d8ff404dae7c9a077fd3d377 2023-04-05 09:56:42,413 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f64647b84b514e639b0b4890259e5119 to hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f64647b84b514e639b0b4890259e5119 2023-04-05 09:56:42,414 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 09:56:42,417 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,419 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 09:56:42,433 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 09:56:42,434 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,434 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 09:56:42,435 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680688602434"}]},"ts":"9223372036854775807"} 2023-04-05 09:56:42,437 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 09:56:42,437 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a168ad47131a50498eee7b0d12023d0b, NAME => 'TestAcidGuarantees,,1680688575500.a168ad47131a50498eee7b0d12023d0b.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 09:56:42,437 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 09:56:42,437 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680688602437"}]},"ts":"9223372036854775807"} 2023-04-05 09:56:42,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 09:56:42,441 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 09:56:42,443 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 09:56:42,443 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-04-05 09:56:42,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 09:56:42,640 INFO [Listener at localhost/42121] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-05 09:56:42,652 INFO [Listener at localhost/42121] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 225) - Thread LEAK? -, OpenFileDescriptor=588 (was 535) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=345 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1057 (was 1457) 2023-04-05 09:56:42,652 INFO [Listener at localhost/42121] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-05 09:56:42,654 INFO [Listener at localhost/42121] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 09:56:42,654 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:62753 2023-04-05 09:56:42,654 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:42,654 DEBUG [Listener at localhost/42121] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-05 09:56:42,654 DEBUG [Listener at localhost/42121] util.JVMClusterUtil(257): Found active master hash=718164389, stopped=false 2023-04-05 09:56:42,654 INFO [Listener at localhost/42121] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:56:42,655 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 09:56:42,655 INFO [Listener at localhost/42121] procedure2.ProcedureExecutor(629): Stopping 2023-04-05 09:56:42,655 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:56:42,655 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 09:56:42,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 09:56:42,656 DEBUG [Listener at localhost/42121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67c1b592 to 127.0.0.1:62753 2023-04-05 09:56:42,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 09:56:42,656 DEBUG [Listener at localhost/42121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:42,656 INFO [Listener at localhost/42121] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46827,1680688416991' ***** 2023-04-05 09:56:42,656 INFO [Listener at localhost/42121] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-05 09:56:42,657 INFO [RS:0;jenkins-hbase4:46827] regionserver.HeapMemoryManager(220): Stopping 2023-04-05 09:56:42,657 INFO [RS:0;jenkins-hbase4:46827] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-05 09:56:42,657 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-05 09:56:42,657 INFO [RS:0;jenkins-hbase4:46827] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-05 09:56:42,657 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(3304): Received CLOSE for d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:56:42,658 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:42,658 DEBUG [RS:0;jenkins-hbase4:46827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x479f9e9d to 127.0.0.1:62753 2023-04-05 09:56:42,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d0f966b2d8313b2cb7f9ce12df1ff570, disabling compactions & flushes 2023-04-05 09:56:42,658 DEBUG [RS:0;jenkins-hbase4:46827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:42,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:56:42,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:56:42,658 INFO [RS:0;jenkins-hbase4:46827] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-05 09:56:42,658 INFO [RS:0;jenkins-hbase4:46827] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-05 09:56:42,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. after waiting 0 ms 2023-04-05 09:56:42,659 INFO [RS:0;jenkins-hbase4:46827] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-05 09:56:42,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:56:42,659 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-05 09:56:42,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d0f966b2d8313b2cb7f9ce12df1ff570 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-05 09:56:42,659 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-05 09:56:42,659 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, d0f966b2d8313b2cb7f9ce12df1ff570=hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570.} 2023-04-05 09:56:42,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 09:56:42,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 09:56:42,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 09:56:42,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 09:56:42,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 09:56:42,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-05 09:56:42,661 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1505): Waiting on 1588230740, d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:56:42,666 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/info/55c40fcad2344f69aeea97bc591fc16a 2023-04-05 09:56:42,670 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55c40fcad2344f69aeea97bc591fc16a 2023-04-05 09:56:42,676 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-05 09:56:42,861 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1505): Waiting on 1588230740, d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:56:43,061 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1505): Waiting on 1588230740, d0f966b2d8313b2cb7f9ce12df1ff570 2023-04-05 09:56:43,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/.tmp/info/98cfba343d32433abc0154350e3b9619 2023-04-05 09:56:43,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/.tmp/info/98cfba343d32433abc0154350e3b9619 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/info/98cfba343d32433abc0154350e3b9619 2023-04-05 09:56:43,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/info/98cfba343d32433abc0154350e3b9619, entries=2, sequenceid=6, filesize=4.8 K 2023-04-05 09:56:43,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d0f966b2d8313b2cb7f9ce12df1ff570 in 414ms, sequenceid=6, compaction requested=false 2023-04-05 09:56:43,075 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/rep_barrier/aede9aea0227420ebfbc9fe060c34b3e 2023-04-05 09:56:43,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/namespace/d0f966b2d8313b2cb7f9ce12df1ff570/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-05 09:56:43,079 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aede9aea0227420ebfbc9fe060c34b3e 2023-04-05 09:56:43,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:56:43,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d0f966b2d8313b2cb7f9ce12df1ff570: 2023-04-05 09:56:43,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680688419173.d0f966b2d8313b2cb7f9ce12df1ff570. 2023-04-05 09:56:43,261 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 09:56:43,462 DEBUG [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 09:56:43,484 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/table/34a84879ea4249a0b6f7b5298b3622b1 2023-04-05 09:56:43,487 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34a84879ea4249a0b6f7b5298b3622b1 2023-04-05 09:56:43,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/info/55c40fcad2344f69aeea97bc591fc16a as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/info/55c40fcad2344f69aeea97bc591fc16a 2023-04-05 09:56:43,490 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55c40fcad2344f69aeea97bc591fc16a 2023-04-05 09:56:43,490 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/info/55c40fcad2344f69aeea97bc591fc16a, entries=22, sequenceid=93, filesize=7.1 K 2023-04-05 09:56:43,491 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/rep_barrier/aede9aea0227420ebfbc9fe060c34b3e as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/rep_barrier/aede9aea0227420ebfbc9fe060c34b3e 2023-04-05 09:56:43,493 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aede9aea0227420ebfbc9fe060c34b3e 2023-04-05 09:56:43,494 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/rep_barrier/aede9aea0227420ebfbc9fe060c34b3e, entries=6, sequenceid=93, filesize=5.5 K 2023-04-05 09:56:43,494 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/.tmp/table/34a84879ea4249a0b6f7b5298b3622b1 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/table/34a84879ea4249a0b6f7b5298b3622b1 2023-04-05 09:56:43,496 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34a84879ea4249a0b6f7b5298b3622b1 2023-04-05 09:56:43,496 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/table/34a84879ea4249a0b6f7b5298b3622b1, entries=9, sequenceid=93, filesize=5.5 K 2023-04-05 09:56:43,497 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 838ms, sequenceid=93, compaction requested=false 2023-04-05 09:56:43,503 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-05 09:56:43,504 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 09:56:43,504 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 09:56:43,504 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 09:56:43,505 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-05 09:56:43,560 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-05 09:56:43,560 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-05 09:56:43,662 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46827,1680688416991; all regions closed. 2023-04-05 09:56:43,670 DEBUG [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/oldWALs 2023-04-05 09:56:43,670 INFO [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46827%2C1680688416991.meta:.meta(num 1680688418889) 2023-04-05 09:56:43,674 DEBUG [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/oldWALs 2023-04-05 09:56:43,674 INFO [RS:0;jenkins-hbase4:46827] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46827%2C1680688416991:(num 1680688418708) 2023-04-05 09:56:43,674 DEBUG [RS:0;jenkins-hbase4:46827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:43,674 INFO [RS:0;jenkins-hbase4:46827] regionserver.LeaseManager(133): Closed leases 2023-04-05 09:56:43,674 INFO [RS:0;jenkins-hbase4:46827] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-05 09:56:43,674 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 09:56:43,676 INFO [RS:0;jenkins-hbase4:46827] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46827 2023-04-05 09:56:43,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46827,1680688416991 2023-04-05 09:56:43,683 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 09:56:43,683 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@696834d2 rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:43,684 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 09:56:43,684 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46827,1680688416991] 2023-04-05 09:56:43,684 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@31d1c8ca rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:43,684 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46827,1680688416991; numProcessing=1 2023-04-05 09:56:43,686 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46827,1680688416991 already deleted, retry=false 2023-04-05 09:56:43,686 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46827,1680688416991 expired; onlineServers=0 2023-04-05 09:56:43,686 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34901,1680688415681' ***** 2023-04-05 09:56:43,686 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-05 09:56:43,686 DEBUG [M:0;jenkins-hbase4:34901] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bf35b9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-05 09:56:43,686 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34901,1680688415681 2023-04-05 09:56:43,686 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34901,1680688415681; all regions closed. 2023-04-05 09:56:43,686 DEBUG [M:0;jenkins-hbase4:34901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 09:56:43,687 DEBUG [M:0;jenkins-hbase4:34901] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-05 09:56:43,687 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-05 09:56:43,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680688418419] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680688418419,5,FailOnTimeoutGroup] 2023-04-05 09:56:43,687 DEBUG [M:0;jenkins-hbase4:34901] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-05 09:56:43,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680688418419] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680688418419,5,FailOnTimeoutGroup] 2023-04-05 09:56:43,687 INFO [M:0;jenkins-hbase4:34901] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-05 09:56:43,687 INFO [M:0;jenkins-hbase4:34901] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-05 09:56:43,687 INFO [M:0;jenkins-hbase4:34901] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-05 09:56:43,687 DEBUG [M:0;jenkins-hbase4:34901] master.HMaster(1512): Stopping service threads 2023-04-05 09:56:43,687 INFO [M:0;jenkins-hbase4:34901] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-05 09:56:43,688 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-05 09:56:43,688 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 09:56:43,688 ERROR [M:0;jenkins-hbase4:34901] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-05 09:56:43,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 09:56:43,688 INFO [M:0;jenkins-hbase4:34901] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-05 09:56:43,688 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-05 09:56:43,689 DEBUG [M:0;jenkins-hbase4:34901] zookeeper.ZKUtil(398): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-05 09:56:43,689 WARN [M:0;jenkins-hbase4:34901] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-05 09:56:43,689 INFO [M:0;jenkins-hbase4:34901] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-05 09:56:43,689 INFO [M:0;jenkins-hbase4:34901] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-05 09:56:43,689 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 09:56:43,689 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:56:43,689 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:56:43,689 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 09:56:43,689 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:56:43,689 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.82 KB heapSize=393.55 KB 2023-04-05 09:56:43,785 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 09:56:43,785 INFO [RS:0;jenkins-hbase4:46827] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46827,1680688416991; zookeeper connection closed. 2023-04-05 09:56:43,785 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@49d91c02 rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:43,786 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): regionserver:46827-0x10153bdea1d0001, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 09:56:43,786 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3dd45f64] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3dd45f64 2023-04-05 09:56:43,786 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@77d2c7fb rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:43,786 INFO [Listener at localhost/42121] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-05 09:56:44,106 INFO [M:0;jenkins-hbase4:34901] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.82 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/75503488c0074024b6af3333fc52c333 2023-04-05 09:56:44,109 INFO [M:0;jenkins-hbase4:34901] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 75503488c0074024b6af3333fc52c333 2023-04-05 09:56:44,110 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/75503488c0074024b6af3333fc52c333 as hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/75503488c0074024b6af3333fc52c333 2023-04-05 09:56:44,113 INFO [M:0;jenkins-hbase4:34901] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 75503488c0074024b6af3333fc52c333 2023-04-05 09:56:44,113 INFO [M:0;jenkins-hbase4:34901] regionserver.HStore(1080): Added hdfs://localhost:45259/user/jenkins/test-data/50bb84d6-67c2-1ad8-6fc9-6fe45d18cdc8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/75503488c0074024b6af3333fc52c333, entries=65, sequenceid=578, filesize=24.6 K 2023-04-05 09:56:44,114 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegion(2948): Finished flush of dataSize ~344.82 KB/353092, heapSize ~393.53 KB/402976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 425ms, sequenceid=578, compaction requested=false 2023-04-05 09:56:44,115 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 09:56:44,115 DEBUG [M:0;jenkins-hbase4:34901] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 09:56:44,117 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 09:56:44,117 INFO [M:0;jenkins-hbase4:34901] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-05 09:56:44,118 INFO [M:0;jenkins-hbase4:34901] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34901 2023-04-05 09:56:44,120 DEBUG [M:0;jenkins-hbase4:34901] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34901,1680688415681 already deleted, retry=false 2023-04-05 09:56:44,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 09:56:44,222 INFO [M:0;jenkins-hbase4:34901] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34901,1680688415681; zookeeper connection closed. 2023-04-05 09:56:44,222 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3abccca7 rejected from java.util.concurrent.ThreadPoolExecutor@a6217fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3994] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:44,222 DEBUG [Listener at localhost/42121-EventThread] zookeeper.ZKWatcher(600): master:34901-0x10153bdea1d0000, quorum=127.0.0.1:62753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 09:56:44,222 ERROR [Listener at localhost/42121-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@57a45577 rejected from java.util.concurrent.ThreadPoolExecutor@a6217fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3994] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 09:56:44,224 WARN [Listener at localhost/42121] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-05 09:56:44,226 INFO [Listener at localhost/42121] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-05 09:56:44,337 WARN [BP-614529572-172.31.14.131-1680688412466 heartbeating to localhost/127.0.0.1:45259] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-05 09:56:44,337 WARN [BP-614529572-172.31.14.131-1680688412466 heartbeating to localhost/127.0.0.1:45259] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-614529572-172.31.14.131-1680688412466 (Datanode Uuid ee8eb019-bae7-4188-8cbb-c2ed540bd4d2) service to localhost/127.0.0.1:45259 2023-04-05 09:56:44,340 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a/dfs/data/data1/current/BP-614529572-172.31.14.131-1680688412466] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 09:56:44,341 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2ccde8d2-8fb0-8754-4f89-4e1cdd9a7740/cluster_7e87272d-f025-e661-09ec-a97f45e13b7a/dfs/data/data2/current/BP-614529572-172.31.14.131-1680688412466] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 09:56:44,403 INFO [Listener at localhost/42121] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-05 09:56:44,566 INFO [Listener at localhost/42121] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-05 09:56:44,736 INFO [Listener at localhost/42121] hbase.HBaseTestingUtility(1293): Minicluster is down